Multichannel audio enhancement, decoding, and rendering in response to feedback

ABSTRACT

In some embodiments, a method for performing at least one of enhancement, decoding, or rendering of a multichannel audio signal in response to compression feedback or feedback from a smart amplifier. For example, the compression feedback may be indicative of amount of compression applied to each of multiple frequency bands, of the audio signal or an enhanced audio signal generated in response thereto. The enhancement (e.g., bass enhancement) may include dynamic routing of audio content of the input audio signal between channels of an enhanced audio signal generated in response thereto. The enhancement and compression may be performed on a per speaker class basis. Other aspects are systems (e.g., programmed processors) and devices (e.g., devices having physically-limited bass reproduction capabilities, such as, for example, a notebook or laptop computer, tablet, soundbar, mobile phone, or other device with small speakers) configured to perform any embodiment of the method.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application claims the benefit of priority to U.S. ProvisionalPatent Application No. 62/827,004, filed Mar. 30, 2019, and U.S.Provisional Patent Application No. 62/688,625, filed Jun. 22, 2018, bothof which are hereby incorporated by reference in their entirety.

TECHNICAL FIELD

The invention pertains to methods and systems for performing enhancement(e.g., bass enhancement) and compression on a multichannel audio signalto generate an enhanced audio signal, and/or decoding and/or renderingon a multichannel audio signal, in response to feedback (e.g.,compression feedback indicative of compression applied to each of atleast two frequency bands of the enhanced audio signal).

BACKGROUND OF THE INVENTION

Speakers have a limited linear region. Many commercially available audioprocessing products employ a broadband limiter to ensure that speakersof (or driven by) the product do not distort. Distortion however is afrequency dependent behavior. To remedy this, a multiband limiter can beemployed. In response to an increase in playback level in a productemploying a multiband limiter, some frequency bands can be unaffected.This can result in a louder sound but also a spectral imbalance in thesound at higher volumes. In some commercially available products, ahybrid approach (known as timbre preservation) is employed that limitsthe audio in a wider band way to ensure the timbre is preserved.

However, conventional approaches to limiting typically do not considerthe previous processing within the audio signal processing chain andtherefore can undo effects that were intentionally applied. ReferencedU.S. Provisional Patent Application 62/688,625 describes methods andsystems which address this problem (and other problems and limitationsof conventional systems) by performing enhancement processing inresponse to compression feedback from a limiter, which can effectivelyprevent the limiting from undoing effects intentionally applied to theaudio.

There are several known methods for modifying an audio signal (therebygenerating an enhanced audio signal) to enhance the low frequency (bass)content perceived during playback of the enhanced audio signal. Thesecan be categorized as:

“equalization-type bass enhancement” technologies which enhance the real(physical) bass response of a speaker employed for playback by boostinglow frequency content via equalization strategies, or

“psychoacoustic bass enhancement” technologies which enhance theperceived bass response of a speaker (e.g., a small loudspeaker)employed for playback via psychoacoustic strategies (e.g., “virtualbass” synthesis or generation methods) designed to increase theperceived level of bass content of an audio signal during playback by atleast one loudspeaker that cannot physically reproduce bass frequenciesof the audio signal.

Equalization strategies are simpler to implement and are generallydeemed to provide a better listening experience than psychoacousticstrategies. Thus, if a speaker (to be employed for playback of an audiosignal) is capable of reproducing real/physical low frequency content,equalization-type bass enhancement is typically applied to the signalrather than psychoacoustic bass enhancement. In some cases (e.g., when aspeaker to be employed for playback is not capable of reproducingreal/physical low frequency content), psychoacoustic bass enhancement isemployed to replace or supplement equalization-type bass enhancement.

At volume levels (of an input audio signal to undergo enhancement andplayback) which are significantly below a speaker's maximum operatinglevel, equalization-type bass enhancement strategies typically workwell. However, at higher volume levels, boosting real/physical lowfrequency content by equalization-type bass enhancement may cause aspeaker to distort at these low frequencies.

It is known to guard against speaker distortion by using a multibandcompressor (e.g., the Audio Regulator of the Dolby Audio API) whichattenuates individual frequency bands of an audio signal according toband energy thresholds which may be configured based on a playbacksystem's real-world distortion characteristics for energy in individualbands. A multiband compressor (sometimes referred to herein as a“regulator” or “multiband limiter”) may limit or attenuate, but does notboost, signal level in any frequency band of the audio signal on whichit operates.

However, equalization-type bass enhancement to boost low frequencycontent may be counteracted (especially when playback at high volumes isintended) by multiband compression (implemented by a regulator) toreduce speaker distortion, sometimes to the point of completecancellation of the bass enhancement. Application of both such bassenhancement and compression may even have the unintended consequence ofreducing overall playback volume, since the regulator may also attemptto preserve timbre (e.g., by not only attenuating at least one band toprevent distortion but also attempting to attenuate neighboring bands bya similar amount).

Psychoacoustic strategies for bass enhancement (e.g., those implementedby “Virtual Bass” processing of the Dolby Audio API) supplement energyfrom lower frequency bands (which the playback speaker is unable toreproduce) with energy in higher frequency bands which the speaker isable to reproduce. Typically this type of bass enhancement processing isused when a speaker is not capable of reproducing low frequency contentat any volume level, due to fundamental physical limitations of thespeaker. However, it could also be used (as it is in some embodiments ofthe present invention) when it is possible but not desirable (e.g., dueto a more nuanced system limitation) for a speaker to reproduce therelevant low frequency content.

One conventional type of psychoacoustic bass enhancement is basssynthesis, which is a collective name for a class of techniques that addin components to the low frequency range of an audio signal in order toenhance the bass that is perceived during playback of the enhancedsignal. Some such techniques (sometimes referred to as sub basssynthesis methods) create low frequency components below the signal'sexisting frequency components in order to extend and improve the lowestfrequency range. Other techniques in the class, known as “virtual pitch”algorithms, generate audible harmonics from an inaudible bass range(e.g., a bass range that is inaudible when the signal is rendered bysmall loudspeakers), so that the generated harmonics improve theperceived bass response. Virtual pitch methods typically exploit thewell-known “missing fundamental” phenomenon, in which low pitches (oneor more low frequency fundamentals, and lower harmonics of eachfundamental) can sometimes be inferred by a human auditory system fromupper harmonics of the low frequency fundamental(s), when thefundamental(s) and lower harmonics (e.g., the first harmonic of eachfundamental) themselves are missing.

BRIEF DESCRIPTION OF THE INVENTION

In a first class of embodiments, the invention is a method for audiosignal compression and enhancement, including steps of: performingenhancement on a multichannel audio signal to generate an enhanced audiosignal; and performing multiband compression on the enhanced audiosignal, thereby generating a compressed, enhanced audio signal, whereinthe enhancement is performed in response to at least one of feedbackfrom a smart amplifier or compression feedback. The compression feedbackis indicative of at least one of: amount of compression applied or to be(e.g., predicted to be) applied to each of at least one frequency band(e.g., each of at least two frequency bands) of (e.g., of at least onechannel of) the enhanced audio signal, or power or amplitude of at leastone channel of the enhanced audio signal, or state of a system volumecontrol. For example, the compression feedback may be indicative ofamount and/or type of distortion (e.g., harmonic and/or intermodulationdistortion) predicted for relevant driver(s) or to be prevented by themultiband compression. The feedback from the smart amplifier may beindicative of at least one of temperature, voltage, current, impedance(e.g., capacitance or inductance), or resistance, of at least onespeaker (e.g., at least one coil of at least one speaker). Theenhancement may include dynamic routing of audio content of the audiosignal between channels of the enhanced audio signal. The dynamicrouting may be frequency dependent.

In some embodiments, the method implements enhancement and compressionof multichannel audio using at least one multichannel approach thatconsiders the power handling characteristics of speakers, for examplespeakers (of a speaker system) that are homogenous in their type and/orspeakers that are heterogeneous in their types. For example, thecompression may be applied by two or more multiband limiters (or two ormore multiband limiters and/or smart amplifiers), each coupled andconfigured to generate compressed audio for playback by a differentsubset of a set of speakers (e.g., where each subset consists ofspeaker(s), of a speaker subsystem, of a different type or class), andthe enhancement may include dynamic routing of audio content of theaudio signal between channels (of the enhanced audio signal) to becompressed (e.g., by different ones of the limiters, or by differentones of the limiters and/or at least one smart amplifier). Typicalembodiments are readily implemented in consumer devices (e.g., soundbarsand televisions, and importantly, in multichannel laptop computers andmultichannel phones). A main goal of typical embodiments is to improvethe volume and/or intelligibility of the system, although this may be atthe expense of spatial fidelity in some embodiments.

In typical embodiments, the enhancement is or includes bass enhancement.Examples of types of enhancement performed in some embodiments include(but are not limited to): bass volume enhancement (e.g., includingrouting of audio content between different channels to undergocompression), virtual bass enhancement, filtering and routing of audiocontent (on a per speaker class basis) according to speaker handlingcharacteristics at the time of playback, dialog enhancement, andvirtualization.

Some embodiments include steps of: enhancing the multichannel audiosignal (e.g., in an enhancement stage or subsystem) to generate theenhanced audio signal; and performing multiband compression (e.g., in aregulator coupled to an output of the enhancement stage or subsystem) onthe enhanced audio signal (e.g., in an effort to prevent distortion uponplayback), where the enhancement is performed in response to thecompression feedback and/or feedback from the smart amplifier.

In another class of embodiments, the invention is a method for audiosignal compression and rendering, including steps of: performingmultiband compression on a multichannel audio signal, thereby generatinga compressed audio signal; and rendering the compressed audio signal inresponse to at least one of feedback from a smart amplifier orcompression feedback. The compression feedback is indicative of at leastone of: amount of compression applied or to be (e.g., predicted to be)applied to each of at least one frequency band (e.g., each of at leasttwo frequency bands) of the multichannel audio signal, or power oramplitude of at least one channel of the multichannel audio signal, orstate of a system volume control. The multichannel audio signal mayinclude speaker channels, or object channels, or both speaker channelsand object channels. The multichannel audio signal may be an enhancedmultichannel audio signal, and the method may also include a step ofperforming (e.g., in response to at least some of the feedback)enhancement on a multichannel audio signal to generate the enhancedmultichannel audio signal.

In other embodiments, the invention is a method for rendering and/ordecoding an audio signal (e.g., a multichannel audio signal) in responseto at least one of compression feedback (e.g., from a multiband limiter)or feedback from a smart amplifier. The rendering (and/or decoding) maybe performed differently in response to different feedback. For example,at least one channel (e.g., each object channel) of the audio signal maybe ignored (i.e., not decoded or not rendered) in response to feedbackindicating that a multiband limiter is limiting at least one band (e.g.,each of at least two bands) of audio content of at least one channel ofthe signal.

Another aspect of the invention is a system (e.g., a device havingphysically-limited or otherwise limited bass reproduction capabilities,such as, for example, a laptop or notebook computer, tablet, soundbar,mobile phone, or other device with small speakers) configured to performany embodiment of the inventive method on an input audio signal.

In a class of embodiments, the invention is an audio playback system(e.g., a notebook, laptop, tablet, soundbar, mobile phone, or otherdevice with (or for use with) small speakers, or a playback system whichhas limited (e.g., physically-limited) bass reproduction capabilities),and is configured to perform audio enhancement (e.g., bass enhancement)on audio in response to compression feedback or feedback from a smartamplifier (in accordance with any embodiment of the inventive method) togenerate enhanced audio, and to playback the enhanced audio.

In some embodiments, the inventive system is a decoder (e.g., configuredto perform audio enhancement in connection with decoding) or an audiorendering system, configured to perform an embodiment of the inventivemethod.

In some embodiments, the inventive system is or includes a general orspecial purpose processor programmed with software (or firmware) and/orotherwise configured to perform an embodiment of the inventive method.In some embodiments, the inventive system is a general purposeprocessor, coupled to receive input audio data, and programmed (withappropriate software) to generate output audio data by performing anembodiment of the inventive method. In some embodiments, the inventivesystem is a digital signal processor, coupled to receive input audiodata, and configured (e.g., programmed) to generate output audio data inresponse to the input audio data by performing an embodiment of theinventive method.

Aspects of the invention include a system configured (e.g., programmed)to perform any embodiment of the inventive method, and a computerreadable medium (e.g., a disc) which stores code for implementing anyembodiment of the inventive method.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a system configured to perform audioenhancement (e.g., bass enhancement).

FIG. 2 is a block diagram of an embodiment of enhancement subsystem 1 ofthe FIG. 1 system.

FIG. 3 is a block diagram of a system configured to perform audioenhancement (e.g., bass enhancement) and compression in accordance withan embodiment of the invention, and rendering of the resultingcompressed, enhanced audio (optionally also in accordance with anembodiment of the invention).

FIG. 3A is a block diagram of a system configured to perform decodingand rendering, and compression and/or amplification, and optionally alsoaudio enhancement (e.g., bass enhancement), in accordance with anembodiment of the invention.

FIG. 4 is a block diagram of a system configured to perform bassenhancement and compression in accordance with an embodiment of theinvention.

FIG. 5 is a block diagram of a system configured to perform bassenhancement (with virtual bass application) and compression inaccordance with an embodiment of the invention.

FIG. 6 is a block diagram of a system configured to perform audioenhancement and compression (on a per speaker class basis) in accordancewith an embodiment of the invention.

FIG. 7 is a block diagram of a system configured to perform audioenhancement (including bass enhancement) and compression (on a perspeaker class basis) in accordance with an embodiment of the invention.

FIG. 8 is a block diagram of a system configured to perform dialogenhancement and compression in accordance with an embodiment of theinvention.

FIG. 9 is a block diagram of a system configured to perform frequencybased dialog enhancement, channel based dialog enhancement, andcompression in accordance with an embodiment of the invention.

FIG. 10 is a block diagram of a system configured to performvirtualization and compression in accordance with an embodiment of theinvention

FIG. 11 is a gain curve for channel distribution of content of a channel(channel “Ls”) of a multichannel audio signal. The horizontal axisindicates each of the channels to which the content is distributed, andthe vertical axis indicates gain applied to the content distributed toeach of the channels.

NOTATION AND NOMENCLATURE

Throughout this disclosure, including in the claims, the expressions“band” and “frequency hand” are used interchangeably, as synonyms.

Throughout this disclosure, including in the claims:

the term channel (or “audio channel”) denotes a monophonic audio signal.Such a signal can typically be rendered in such a way as to beequivalent to application of the signal directly to a loudspeaker at adesired or nominal position. The desired position can be static, as istypically the case with physical loudspeakers, or dynamic;

“speaker channel” denotes an audio channel that is associated with anamed loudspeaker (at a desired or nominal position), or with a namedspeaker zone within a defined speaker configuration. A speaker channelis rendered in such a way as to be equivalent to application of theaudio signal directly to the named loudspeaker (at the desired ornominal position) or to a speaker in the named speaker zone;

“object” (or “audio object” or “object channel”) denotes an audiochannel indicative of sound emitted by an audio source (sometimes alsoreferred to as an audio “object”). Typically, an object channeldetermines a parametric audio source description (e.g., metadataindicative of the parametric audio source description is included in orprovided with the object channel). The source description may determinesound emitted by the source (as a function of time), the apparentposition (e.g., 3D spatial coordinates) of the source as a function oftime, and optionally at least one additional parameter (e.g., apparentsource size or width) characterizing the source;

“channel” of a multichannel audio signal denotes either a speakerchannel or an audio object (or the audio content, excluding relatedmetadata, of an audio object);

“object-based audio” denotes audio (e.g., an audio signal or audioprogram) comprising a set of one or more object channels (and optionallyalso comprising at least one speaker channel) and optionally alsoassociated metadata (e.g., metadata indicative of a trajectory of anaudio object which emits sound indicated by an object channel, ormetadata otherwise indicative of a desired spatial audio presentation ofsound indicated by an object channel, or metadata indicative of anidentification of at least one audio object which is a source of soundindicated by an object channel); and

“render” denotes the process of converting audio (e.g., an audioprogram) into one or more speaker feeds, or the process of convertingaudio (e.g., an audio program) into one or more speaker feeds andconverting the speaker feed(s) to sound using one or more loudspeakers(in the latter case, the rendering is sometimes referred to herein asrendering “by” the loudspeaker(s)).

Throughout this disclosure, including in the claims, the expression“multiband compression” of or on an audio signal (e.g., on frequencydomain data indicative of an enhanced audio signal or other audiosignal, or on one or more channels of a multi-channel audio signal)denotes limiting compression on a band-by-band basis (in at least twodifferent frequency bands) which does not increase level of the signalin any frequency band. In each band, multiband compression eitherreduces (or does not change, or does not change by a substantial orsignificant amount) level of the signal. Multiband compression issometimes referred to herein as “regulation,” and a compressor whichperforms or is configured to perform multiband compression is sometimesreferred to herein as a “regulator.”

Throughout this disclosure, including in the claims, the expression“enhancement” (or “audio enhancement”) of or on an audio signal (e.g.,on frequency domain data indicative of an audio signal, or one or morechannels of a multi-channel audio signal) denotes any enhancingoperation performed on the signal. For example, enhancement may be anenhancing operation performed on a band-by-band basis (in at least twodifferent frequency bands of the signal) on the signal. Examples ofaudio enhancement include, but are not limited to, bass enhancement(e.g., equalization-type bass enhancement or psychoacoustic bassenhancement), dialog enhancement, upmixing, frequency shifting, harmonicinjection or transposition, subharmonic injection, virtualization, andequalization.

Throughout this disclosure, including in the claims, the expressionperforming an operation “on” a signal or data (e.g., filtering, scaling,transforming, or applying gain to, the signal or data) is used in abroad sense to denote performing the operation directly on the signal ordata, or on a processed version of the signal or data (e.g., on aversion of the signal that has undergone preliminary filtering orpre-processing prior to performance of the operation thereon).

Throughout this disclosure including in the claims, the expression“system” is used in a broad sense to denote a device, system, orsubsystem. For example, a subsystem that implements a decoder may bereferred to as a decoder system, and a system including such a subsystem(e.g., a system that generates X output signals in response to multipleinputs, in which the subsystem generates M of the inputs and the otherX−M inputs are received from an external source) may also be referred toas a decoder system.

Throughout this disclosure including in the claims, the term “processor”is used in a broad sense to denote a system or device programmable orotherwise configurable (e.g., with software or firmware) to performoperations on data (e.g., audio, or video or other image data). Examplesof processors include a field-programmable gate array (or otherconfigurable integrated circuit or chip set), a digital signal processorprogrammed and/or otherwise configured to perform pipelined processingon audio or other sound data, a programmable general purpose processoror computer, and a programmable microprocessor chip or chip set.

Throughout this disclosure including in the claims, the term “couples”or “coupled” is used to mean either a direct or indirect connection.Thus, if a first device couples to a second device, that connection maybe through a direct connection, or through an indirect connection viaother devices and connections.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Many embodiments of the present invention are technologically possible.It will be apparent to those of ordinary skill in the art from thepresent disclosure how to implement them. Embodiments of the inventivesystem, method, and medium will be described with reference to FIGS. 3,3A, and 4-11.

FIG. 1 is a block diagram of a system (9) configured to perform audioenhancement. In the FIG. 1 system, enhancement subsystem 1 is coupledand configured to perform audio enhancement on an input audio signal,thereby generating an enhanced audio signal. Regulator 3 (sometimesreferred to as multiband compression subsystem 3) is coupled toenhancement subsystem 1 and configured to perform multiband compressionon the enhanced audio signal, thereby generating an output audio signalwhich is a compressed, enhanced audio signal. In operation, subsystem 3applies compression on a band-by-band basis to the enhanced audio signal(i.e., to reduce or leave unchanged the level of each frequency band, ateach time of a sequence of times, of the enhanced audio signal) in amanner intended to prevent distortion upon playback of the compressed,enhanced audio signal output from subsystem 3. Subsystem 3 is alsoconfigured to generate a compression signal indicative of the amount ofcompression (level attenuation) applied by subsystem 3 to each of atleast one frequency band (e.g., to each of at least two individualfrequency bands, or to each of the entire set of individual frequencybands) of the enhanced audio signal, and this compression signal isprovided as a feedback to enhancement subsystem 1. The compressionsignal is thus a feedback signal indicative of amount of compressionapplied by regulator subsystem 3 to at least one frequency band (or eachof at least two frequency bands) of the enhanced audio signal.

Enhancement subsystem 1 is configured to perform audio enhancement inresponse to the compression signal (a feedback signal indicative ofamount of compression applied to at least one frequency band (e.g., eachof at least two individual frequency bands) of the enhanced audiosignal.

The FIG. 1 system also includes mixed signal processing subsystem 5(coupled to regulator 3) and speaker 7 (coupled to subsystem 5). Inoperation, the compressed, enhanced audio signal output from regulator 3is provided to subsystem 5, and subsystem 5 (with speaker 7) performplayback of audio content of the compressed, enhanced audio signal.Subsystem 5 is configured to generate a speaker feed in response to thecompressed, enhanced audio signal. The speaker feed is provided tospeaker 7, and speaker 7 is configured to emit sound in response to thespeaker feed. Typically, the compression performed by regulator 3prevents the sound from being distorted.

Thus, subsystem 5 is configured to render audio content (indicated bythe compressed, enhanced audio signal) by converting the content intothe speaker feed of the compressed, enhanced audio signal (and thussubsystem 5 may be referred to as a rendering subsystem), and subsystem5 (together with speaker 7) is configured to render such audio contentby converting the content into the speaker feed and converting thespeaker feed to sound.

In some embodiments, enhancement subsystem 1 of FIG. 1 is configured toperform one or both of psychoacoustic bass enhancement (e.g., harmonictransposition) and equalization-type bass enhancement to generate a bassenhanced audio signal in response to the input audio signal, and thebass enhancement is performed in response to feedback indicative ofamount of compression applied by a regulator (e.g., regulator 3) to eachof at least two frequency bands of the bass enhanced audio signal (e.g.,to selectively apply one or the other, or both, of the two types of bassenhancement to the input audio signal) in a manner controlled by thefeedback. FIG. 2 is a block diagram of an example of such an embodimentof enhancement subsystem 1 of the FIG. 1 system.

In FIG. 2, transform element 6 is configured to perform a timedomain-to-frequency domain transform (e.g., FFT) on an input audiosignal to generate banded input audio indicative of the audio content ofthe input audio signal, such that the banded input audio includes asequence of frequency components for each different frequency band of aset of frequency bands. Thus, in the FIG. 2 embodiment (as in someembodiments of the invention) the input data to be enhanced are bandedfrequency domain audio data indicative of the audio content of the inputaudio signal. Psychoacoustic bass enhancement (“PBE”) subsystem 8 iscoupled and configured to perform psychoacoustic bass enhancement on thebanded input audio (typically, including by boosting (increasing thelevel of) content in frequency bands other than the lowest frequencybands). Equalization-type bass enhancement subsystem (“equalizer”) 10 iscoupled and configured to perform equalization-type bass enhancement onthe banded input audio (typically, to boost content in low frequencybands). Combination subsystem 12 is coupled to receive the audio outputof each of subsystems 8 and 10, and the compression feedback signal(“compression feedback”) generated by regulator 3 of FIG. 1, and isconfigured to generate, in response thereto, banded enhanced audio. Thebanded enhanced audio is the output of the FIG. 2 embodiment ofenhancement subsystem 1 and is provided to regulator 3 of FIG. 1. Thecompression feedback is indicative of amount of compression applied byregulator 3 to each of at least two frequency bands of the bandedenhanced audio. Typically, the frequency bands of the banded enhancedaudio (output from combination subsystem 12) are the same as those towhich regulator 3 applies compression (and the same as those of thebanded audio which is output from transform in subsystem 6), and thecompression feedback is indicative of amount of compression applied byregulator 3 to each of these bands.

Combination subsystem 12 is configured to combine the outputs ofsubsystems 8 and 10 to generate the banded enhanced audio to be (at anytime): the output of one or the other of subsystems 8 and 10 for thetime; or a combination (e.g., linear combination) of frequencycomponents output from subsystems 8 and 10 for the time.

In other embodiments (not shown in FIG. 2), subsystems 8 and 10 arecoupled in series, and subsystem 12 is omitted.

Combination subsystem 12 is typically configured to generate the bandedenhanced audio as a sequence of banded enhanced audio values, with thebanded enhanced audio values corresponding to each time (or timeinterval) consisting of values for each of a number of differentfrequency bands, and such that each of the values for one time (or timeinterval) and one band is:

a combination (e.g., linear combination) of frequency components outputfrom subsystems 8 and 10 for the time (or time interval) and band (e.g.,in response to some values of the compression feedback for thecorresponding time or interval), or

a frequency component output from one or the other of subsystems 8 and10 for the time (or time interval) and band (e.g., in response to someother values of the compression feedback for the corresponding time orinterval).

For example, when the compression feedback indicates that regulator 3 isnot applying compression in any band, the output of subsystem 12 foreach band (at the corresponding time or time interval) may be afrequency component output from subsystem 10. If the compressionfeedback (corresponding to a later time or time interval) then indicatesthat regulator 3 is applying compression in each band (to preventdistortion), the output of subsystem 12 for each band (at thecorresponding time or interval) may be a frequency component output fromsubsystem 8.

For another example, when the compression feedback indicates thatregulator 3 is not applying compression (or is applying a small amountof attenuation) in a band, the output of subsystem 12 for the band (atthe corresponding time or time interval) may be a first linearcombination of a frequency component output from subsystem 8 and afrequency component output from subsystem 10 (e.g., aX+bY, where a and hare factors, X is the frequency component output from subsystem 8, and Yis the frequency component output from subsystem 10). If the compressionfeedback (corresponding to a later time or time interval) then indicatesthat regulator 3 is applying compression (or is applying a greateramount of attenuation) in the band, the output of subsystem 12 for eachband (at the corresponding time or interval) may be a second linearcombination (different than the first linear combination) of a frequencycomponent output from subsystem 8 and a frequency component output fromsubsystem 10 (e.g., cX+dY, where c is a factor different than a, d is afactor different than b, X is the frequency component output fromsubsystem 8, and Y is the frequency component output from subsystem 10).

Alternatively (or additionally), the compression feedback is provided tosubsystem 8 and/or subsystem 10 (as indicated by dashed lines in FIG. 2)to control the manner in which subsystem 8 and/or subsystem 10 performsbass enhancement. For example, operation of one or both of subsystems 8and 10 may be enabled or disabled (during a time interval) in responseto the compression feedback, and/or the manner in which one or both ofsubsystems 8 and 10 performs enhancement on the banded input audio maybe controlled by the compression feedback.

For example, PBE subsystem 8 may perform harmonic transposition usingeven harmonics in response to some values of the compression feedback,and/or using odd harmonics in response to some other values of thecompression feedback. In typical operation of the FIG. 2 system, bassenhancement by subsystem 8 and/or subsystem 10 is controlled (by thecompression feedback) so as to prevent the bass enhancement from causingdistortion (upon playback) in any specific band (given the amount ofcompression being applied by regulator 3 in that band), and/or toprevent regulator 3 from applying compression in one or more specificbands. Less (or a different type of) processing by subsystem 8 (orsubsystem 10) in a band may be desired in the case that regulator 3 isapplying a lot of attenuation in the band, to prevent distortion.

For example, a boost (applied by subsystem 10) in a band may be reducedin the case that the feedback indicates that regulator 3 applies a lotof attenuation (e.g., attenuation in excess of a predetermined thresholdamount) in the band, e.g., in cases in which too much boosting (bysubsystem 10) with high compression (by regulator 3) could result indistortion. In some embodiments, an amount or degree of processing byone of subsystems 8 or 10 is determined in response to the amount ordegree of processing by the other one of subsystems 8 or 10 (which is inturn determined by the compression feedback), e.g., to keep a totalamount or degree of processing by both subsystems 8 and 10 constant orat a desired amount.

In typical implementations of system 9 of FIG. 1, a full bandwidthcompression feedback signal (indicative of compression applied byregulator 3 in each band of a full set of bands) is fed into enhancementsubsystem 1 (sometimes referred to as an enhancement layer). Inresponse, enhancement subsystem 1 produces an enhanced audio signal thatis fed into regulator 3, and regulator 3 and subsystem 5 operate toproduce a non-distorted speaker feed. The compression feedback signalmay have some gating to ensure there is no undesirable pumping of thesignal output from regulator 3.

Typical reasons for a change in the amount of limiting (attenuation)applied by regulator 3 (in at least one band) include a change inplayback volume due to a user control or a change in level of an audiosignal provided to or generated by the system. It is fundamentallyimportant that the regulator be placed in series after the enhancementlayer (in the sense that the regulator operates on the output of theenhancement layer) to ensure that the speaker is not fed with a signalthat would cause it to distort.

With reference to FIG. 3, a first class of embodiments (e.g., system100, and a system comprising elements 100, 108, and 109 of FIG. 3) ofthe inventive method and system will be described. System 100 includesmultichannel processing subsystem 102 which is coupled and configured togenerate audio signal 103 (which includes one or more channels, andwhich will sometimes be referred to herein as “enhanced audio signal”103) in response to multichannel audio input signal 101 (which includesat least two channels). Typically, the channels of signal 101 (andsignal 103) are speaker channels. In some other embodiments (e.g., asdescribed with reference to FIG. 3A, and elsewhere herein) the channelsof the audio input signal are object channels or include at least oneobject channel and at least one speaker channel. When signal 101includes at least one speaker channel, the number of speaker channels ofsignal 101 may equal the number of speaker channels of signal 103 (andthe number of speaker feeds output from subsystem 108), or the number ofspeaker channels of signal 101 may be less than (e.g., when upmixing isimplemented) or greater than the number of speaker channels of signal103 (and/or the number of speaker feeds output from subsystem 108).Signal 101 may be the output of a decoder (e.g., as is the output ofdecoder 802 of FIG. 3A). In some cases, system 100 (e.g., subsystem 102thereof) may perform decoding (or at least one step of decoding) onsignal 101.

System 100 also includes multiband limiter 104 which is coupled tosubsystem 102 and configured to perform compression on frequency bandsof at least one (and typically each) channel of enhanced audio signal103, to generate compressed, enhanced audio output signal 106. Multibandlimiter 104 is also configured to generate compression feedback signal105 which is typically indicative of an amount of compression applied bylimiter 104 in each of at least one frequency band (e.g., each of atleast two frequency bands) of each of at least one channel of signal103. Subsystem 102 is coupled and configured to receive compressionfeedback 105 from limiter 104, and to perform multichannel enhancementor other multichannel audio-based processing (e.g., rendering or a stepof rendering, and/or decoding or a step of decoding) on audio inputsignal 101 in response to compression feedback 105, to generate enhancedaudio signal 103. For example, enhancement performed by subsystem 102may be or include bass enhancement, or another type of enhancement(e.g., any of those described herein).

In operation, limiter 104 performs multiband compression on signal 103to generate output signal 106 (which may include one or more channels)for playback (or rendering and then playback) by a speaker systemincluding at least one speaker. In some implementations (e.g., those tobe described with reference to FIGS. 6 and 7), limiter 104 is replacedby two or more multiband limiters, each configured to generate audiooutput for playback by a different subset of the speakers of amulti-speaker implementation of a speaker subsystem (e.g., subsystem109).

Optionally (e.g., in cases in which subsystem 102 implementsmultichannel audio-based processing other than rendering on audio inputsignal 101), the FIG. 3 system also includes mixed signal processingsubsystem 108 and speaker subsystem 109 (coupled to subsystem 108).Subsystem 108 is coupled to limiter 104 (or each limiter of animplementation in which limiter 104 is replaced by two or more limiters)to receive output 106 of limiter 104 (or each limiter of a set oflimiters which replaces limiter 104). In typical operation, thecompressed, enhanced audio signal 106 output from limiter 104 isprovided to subsystem 108, and subsystem 108 (with speaker subsystem109) performs playback of audio content of the compressed, enhancedaudio signal 106. Subsystem 108 is configured to generate a speaker feedfor each speaker of subsystem 109 in response to (e.g., by performingdigital-to-analog conversion on) the compressed, enhanced audio signal106 (and thus subsystem 108 may be referred to as a renderingsubsystem). Each speaker feed is provided to speaker subsystem 109, andspeaker subsystem 109 is configured to emit sound in response to eachspeaker feed. Typically, the compression performed by limiter 104 (oreach limiter employed in place of limiter 104) prevents the sound frombeing distorted.

Thus, subsystem 108 is configured to render audio content (indicated bythe compressed, enhanced audio signal) by converting the content into atleast one speaker feed, and subsystem 108 (together with speakersubsystem 109) is configured to render such audio content by convertingthe content into at least one speaker feed and converting each speakerfeed to sound. In cases in which audio input signal 101 includes speakerchannels (but no object channel), the rendering performed by subsystem108 may include or consist of digital-to-analog conversion (andoptionally also amplification) of each speaker channel output fromlimiter 104, to generate a speaker feed for each such speaker channel.In cases in which audio input signal 101 includes at least one objectchannel (e.g., as do typical examples of audio input signal 801 of FIG.3A), subsystem 102 (or another system element not specifically shown inFIG. 3) may be configured perform object-based rendering (whichtypically includes mixing of audio content of object channel(s) tospeaker channel(s)), and subsystem 108 may perform a rendering stepconsisting of (or including) digital-to-analog conversion of eachspeaker channel output from limiter 104, to generate a speaker feed foreach such speaker channel.

Element 100 (and optionally also subsystem 108) of FIG. 3 may be aprocessor which is programmed (or otherwise configured) to perform anembodiment of the inventive method, and in which elements 102 and 104(and optionally also subsystem 108) of FIG. 3 are implemented assubsystems (e.g., stages) of the processor. For example, elements 100and 108 of FIG. 3 are implemented in a decoder or other audio processor(e.g., a device including a decoder and a post-processor coupled toprocess output of the decoder), or other device configured to performaudio signal processing. In another example, elements 100, 108, and 109of FIG. 3 are implemented as a playback device (e.g., laptop computer)configured to perform an embodiment of the inventive method, and inwhich elements 102, 104, 108, and 109 of FIG. 3 are implemented assubsystems (e.g., stages) of the playback device.

Typically, multiband limiter 104 (or each limiter employed in place ofit) has been tuned to limit when each speaker would distort. Feedbacksignal 105 is typically is indicative of compression (gain) applied bylimiter 104 (or each limiter employed in place of it) to each of a setof frequency bands (e.g., each band of a full set of frequency bands) ofeach channel (or each of one or more of the channel(s)) of signal 103.More generally, compression feedback 105 is indicative of amount ofcompression applied or to be (e.g., predicted to be) applied to each ofat least one frequency band (e.g., each of at least two frequency bands)of (e.g., of at least one channel of) the enhanced audio signal, orpower or amplitude of at least one channel of the enhanced audio signal,or state of a system volume control. For example, the compressionfeedback may be indicative of amount and/or type of distortion (e.g.,harmonic and/or intermodulation distortion) predicted for relevantdriver(s) or to be prevented by the multiband compression.Alternatively, compression feedback 105 is not provided by limiter 104,and/or is replaced by feedback from a smart amplifier (e.g., asdescribed with reference to the FIG. 3A embodiment).

In typical operation of subsystem 102 in response to feedback 105,subsystem 102 changes the processing (e.g., enhancement) it performs oninput signal 101 in response to the amount of compression (indicated byfeedback 105) applied to at least one channel (e.g., each channel) ofsignal 103 by limiter 104 (or each limiter employed in place of it) ineach of one or more frequency bands.

Next, with reference to FIG. 3A, another class of embodiments of theinventive method and system will be described. System 800 of FIG. 3Aincludes decoding subsystem 802 (e.g., a decoding subsystem configuredto implement Dolby Digital Plus decoding) on multichannel audio inputsignal 801. Signal 801 includes at least two channels of encoded audiocontent. Typically, the channels are object channels, or include atleast one object channel and at least one speaker channel.

System 800 also includes rendering subsystem 803, enhancement subsystem805, multiband limiter 806, and smart amplifier 808, coupled as shown.Optionally, one or more of subsystems 802, 803, 805, 806, or 808, is/areomitted (with the constraint that at least one of subsystems 802, 803,or 805, and at least one of elements 806 or 808, is present and at leastone of subsystems 802, 803, or 805 is provided with feedback 807 and/orfeedback 809. For example, optionally, subsystem 805 and limiter 806 areomitted (and the output 804 of subsystem 803 is provided to amplifier808), or amplifier 808 is omitted. Optionally, multiband limiter 806 isreplaced by two or more limiters (e.g., two or more limiters as in theembodiment of FIG. 6 or 7). Typically, the audio signal which is outputfrom limiter 806 or amplifier 808 (or one or more speaker feedsgenerated in response thereto) is provided to a speaker system (e.g., aspeaker system identical to speaker subsystem 109 of FIG. 3), forplayback of audio content of the audio signal. Typically, thecompression performed by limiter 806 (or each limiter employed in placeof limiter 806) prevents the sound emitted from the speaker system frombeing distorted.

Rendering subsystem 803 is coupled and configured to performobject-based rendering on the decoded audio channels output fromsubsystem 802, to generate one or more speaker channels 804 (sometimesreferred to herein as signal 804). For example, subsystem 803 maygenerate N speaker channel(s) in response to M object channels (orobject channels and speaker channels) generated by subsystem 802 inresponse to input signal 801, where N and M are numbers. Another elementof system 800 (or a system coupled to system 800) may also performdigital-to-analog conversion (and optionally also amplification) of eachspeaker channel output from limiter 806 or amplifier 808, to generate aspeaker feed for each such speaker channel. For example, the output(signal 811) of limiter 806 (implemented in software as a programmedprocessor) may undergo digital to analog conversion, followed byamplification (dumb, rather than smart, amplification) to generatespeaker feed(s). For another example, smart amplifier 808 may implementsmart amplification including digital-to-analog conversion on the outputof limiter 806, so that the output (signal 813) of smart amplifier 808is a set of one or more speaker feeds. Smart amplifier 808 may includestructure similar to that of a software implementation of limiter 806,but the limiting implemented by amplifier 808 is typically driven by itsexcursion protection model, and amplifier 808 is wired directly (duringuse) to the speaker(s) since amplifier 808 provides analogamplification, and analog sensing and/or measurement.

Enhancement subsystem 805 is coupled and configured to generate audiosignal 810 (which includes one or more channels, and which willsometimes be referred to herein as “enhanced audio signal” 810) inresponse to signal 804. Subsystem 805 may be implemented to perform (onsignal 804) enhancement (e.g., multichannel enhancement) of any of thetypes described herein (e.g., of any of the types performed by anembodiment of subsystem 102 of FIG. 3).

Multiband limiter 806 which is coupled to subsystem 805 and configuredto perform compression on frequency bands of at least one (and typicallyeach) channel of enhanced audio signal 810, to generate compressed,enhanced audio signal 811. Multiband limiter 806 is also configured togenerate compression feedback signal 807 which is typically indicativeof an amount of compression applied by limiter 806 in each of at leastone frequency band (e.g., each of at least two frequency bands) of eachof at least one channel of signal 810. Subsystem 805 is coupled andconfigured to receive compression feedback 807 from limiter 806, and toperform enhancement (e.g., multichannel enhancement) on signal 804 inresponse to compression feedback 807, to generate enhanced audio signal810. For example, enhancement performed by subsystem 805 may be orinclude bass enhancement, or another type of enhancement (e.g., any ofthose described herein). Subsystems 802 and 803 are typically alsocoupled and configured to receive feedback 807, and to perform decoding(or a step of decoding) or rendering (or a step of rendering) inresponse to feedback 807.

In operation, limiter 806 performs multiband compression on signal 810to generate signal 811 (which may include one or more channels) forplayback (or digital-to-analog conversion and then playback) by aspeaker system including at least one speaker. In some implementations(e.g., those described with reference to FIGS. 6 and 7), limiter 806 isreplaced by two or more multiband limiters, each configured to generateaudio output for playback by a different subset of the speakers of amulti-speaker implementation of a speaker subsystem (e.g., subsystem109).

Smart amplifier 808 is coupled and configured to perform smartamplification on signal 811 to generate signal 813. Signal 813 (whichmay include one or more channels) may be provided to a speaker system(including at least one speaker) for playback.

Amplifier 808 may be configured to implement analog to digitalconversion and amplification, e.g., to generate a digital input. In someimplementations, the input to amplifier 808 is an analog input, andamplifier 808 is configured to perform analog-to digital conversion,followed by processing, followed by digital-to-analog conversion, andfinally amplification. In all cases, amplifier 808 is configured toperform amplification. Amplifier 808 is typically connected (wired)directly to a speaker subsystem, in order to measure the voltage and/orcurrent and/or impedance of each driver to which it is connected.

Smart amplifier 808 is also configured to generate feedback signal 809,which may be of any of the types of smart amplifier feedback describedherein. At least one (e.g., all) of subsystems 802, 803, and 805 is (orare) coupled and configured to receive feedback 809 from amplifier 808,and to perform decoding (or a step of decoding), rendering (or a step ofrendering), or multichannel enhancement in response to feedback 809. Forexample, subsystem 802 may be coupled and configured to receive feedback809 from amplifier 808 and to perform decoding (or a step of decoding)on signal 801 in response thereto. Subsystem 803 may be coupled andconfigured to receive feedback 809 from amplifier 808 and to performrendering (or a step of rendering) on the output of subsystem 802 inresponse thereto, to generate signal 804. Subsystem 805 may be coupledand configured to receive feedback 809 from amplifier 808 and to performmultichannel enhancement in response thereto to generate signal 810.

In some implementations, smart amplifier feedback 809 is provided tolimiter 806, e.g., for use by limiter 806 to adjust the excitations oflimiter 806. In at least some such implementations; compression feedback807 from limiter 806 may be indicative of feedback 809 (or feedback 807may be indicative of at least one quantity related to feedback 809).

System 800 of FIG. 3A (or one or more of subsystems 802, 803, 805, 806,and 808 of system 800) may be a processor which is programmed (orotherwise configured) to perform an embodiment of the inventive method.For example, subsystems 802, 803, 805, 806, and/or 808 of system 800 maybe implemented as subsystems (e.g., stages) of the processor. In oneexample, system 800 may be implemented in a decoder or other audioprocessor (e.g., a device including a decoder and a post-processorcoupled to process output of the decoder), or other device configured toperform audio signal processing. In another example, system 800 isimplemented as a playback device (e.g., laptop computer) configured toperform an embodiment of the inventive method, and in which elements802, 803, 805, 806, and 808 of FIG. 3A are implemented as subsystems(e.g., stages) of the playback device.

Typically, multiband limiter 806 (or each limiter employed in place ofit) has been tuned to limit when each speaker would distort. Feedbacksignal 807 is typically indicative of compression (gain) applied bylimiter 806 (or each limiter employed in place of it) to each of a setof frequency bands (e.g., each band of a full set of frequency bands) ofeach channel (or each of one or more of the channel(s)) of signal 810.More generally, compression feedback 807 is indicative of amount ofcompression applied or to be (e.g., predicted to be) applied to each ofat least one frequency band (e.g., each of at least two frequency bands)of (e.g., of at least one channel of) the enhanced audio signal, orpower or amplitude of at least one channel of the enhanced audio signal,or state of a system volume control. For example, the compressionfeedback may be indicative of amount and/or type of distortion (e.g.,harmonic and/or intermodulation distortion) predicted for relevantdriver(s) or to be prevented by the multiband compression.

In some embodiments, the inventive system (e.g., subsystem 100 of FIG.3, or system 800 of FIG. 3A or at least one subsystem of system 800) isconfigured to perform enhancement (other than, or in addition to, bassenhancement) to generate an enhanced audio signal (in response to amultichannel input audio signal) using compression feedback and/orfeedback from a smart amplifier. We next describe examples of some typesof such enhancement (e.g., performed by embodiments of subsystem 100 ofFIG. 3 or system 800 of FIG. 3A) that can be controlled usingcompression feedback (and/or feedback from a smart amplifier) in someembodiments of the invention. The examples include:

1. Dialog Enhancement

When performing dialog enhancement (e.g., by operating an embodiment ofsubsystem 102 of FIG. 3 or subsystem 805 of FIG. 3A), the level of adialog enhancement signal (e.g., generated by the embodiment ofsubsystem 102) may be decreased in response to compression feedback(e.g., from regulator 104) to limit the maximum level (in one or morespecific bands) of the dialog-enhanced audio signal which is provided tothe regulator, to cause such maximum level to be sufficiently low toprevent the regulator from compressing (limiting) audio in such band(s).If the level of the dialog enhancement signal is not so decreased whenthe regulator is limiting the dialog-enhanced audio signal (in at leastone band), the dialog enhancement will often make the dialog (indicatedby the compressed, dialog-enhanced audio signal output from theregulator) harder to understand rather than more intelligible.

In some embodiments, the shape of the dialog enhancement curve (employedto perform dialog enhancement) may be changed in response to thecompression feedback, to reduce the gain of the dialog-enhanced audiosignal in each band (which is outside the typical speech frequencyrange, i.e., 300-3000 Hz, and is being compressed by the regulator asindicated by the compression feedback), to prevent the regulator (e.g.,regulator 104) from continuing to apply compression in each such band ofthe dialog-enhanced audio signal. For example, when subsystem 102 isconfigured to perform the dialog enhancement, the gain of subsystem102's output in each band within the speech frequency range wouldtypically not be reduced (but the gain of subsystem 102's output in eachband outside the speech frequency range would in some cases be reduced)in response to the compression feedback. This can be done to ensure thata timbre preservation mode of the regulator (e.g., regulator 104) doesnot result in a compressed, dialog-enhanced audio signal (output fromthe regulator) that has dialog that is too quiet and still ensure thatincreases in user-controlled volume result in an increase in dialogvolume;

2. Upmixing

When performing upmixing (e.g., by operating an embodiment of subsystem102 of FIG. 3 or subsystem 805 of FIG. 3A), the amount of diffusecontent (e.g., generated by the embodiment of subsystem 102) may bereduced (while keeping the direct content untouched) in response tocompression feedback (e.g., from regulator 104), when the compressionfeedback indicates that the regulator is limiting relevant bands of theupmixed audio signal (i.e., relevant bands of at least one channel ofmulti-channel upmixed audio), to reduce the amount of energy of theupmixed audio signal that is fed into the regulator. Alternatively, theupmixing may be disabled (so that no upmixing is performed at all) inresponse to the compression feedback for specific time intervals inwhich the compression feedback indicates that this should be done;

3. Volume leveling, modeling, or automatic gain control (e.g., asimplemented by Dolby Volume). When performing volume modeling (e.g., byoperating an embodiment of subsystem 102 of FIG. 3 or subsystem 805 ofFIG. 3A), the volume modeler may analyze incoming audio, group similarfrequencies into critical bands, and apply appropriate amounts of gainto each band, in a manner controlled by compression feedback (e.g., fromregulator 104) which compresses the output of the volume modeler. Inresponse to the compression feedback, the volume modeler may adjust thefrequency response for different playback levels (relative to an assumedreference level, which is typically around 85 decibels) to compensatefor the way people perceive audio during playback at different playbacklevels. Thus, the volume modeling may assure that a user always hearsthe correct tonal balance, whether at high or low playback levels.

When performing volume leveling (e.g., by operating an embodiment ofsubsystem 102), the volume leveler may operate in a manner controlled bycompression feedback (e.g., from regulator 104) which compresses theoutput of the volume leveler. The volume leveler may control playbacklevel of input audio to maintain consistent playback level regardless ofthe source selection and content.

In some examples of implementations of subsystem 100 (or system 800),the enhancement subsystem may be controlled in response to compressionfeedback in any of the following ways:

a targeted reference level of a volume leveler or a reference level of avolume modeler (implemented by subsystem 102 or 805) may be adjusted inresponse to the compression feedback to ensure that subsystem 102 (or805) is not driving (e.g., continuously driving) regulator 104 (or 806)to cause the regulator to compress audio in one or more specific bands.The targeted reference level may be calibrated using the regulatortuning; or

the gain swing of automatic gain control (AGC) implemented by subsystem102 (or 805) may be adjusted in response to the compression feedback tolimit the maximum level of the output of subsystem 102 or 805 (in one ormore specific bands) is sufficiently low to prevent regulator 104 (or806) from compressing audio in such band(s);

4. Frequency Shift Block

To increase the speech intelligibility (e.g., of audio captured duringconference calls), subsystem 102 (or 805) may be implemented as (or toinclude) a frequency shift block. When operating such an embodiment ofsubsystem 102, the frequency shift block may operate in a mannercontrolled by compression feedback (e.g., from regulator 104) whichcompresses the output of the frequency shift block. Typically, as a userincreases the volume and the regulator starts limiting the frequencybands in the range of typical speech, the frequency shift block willshift all the frequencies in a direction that would give an increase inperceived volume taking into account the capabilities of the playbackdevice (and optionally the noise level in the surrounding environment);

5. Harmonic Injection

In the case where bands are being limited by a regulator, compressionfeedback from the regulator can be provided to an embodiment ofsubsystem 102 (or 805). The subsystem can operate in response to thecompression feedback to inject harmonic psychoacoustic frequencies intoan audio input signal (e.g., to provide virtual bass) and therebygenerate an enhanced signal which is provided to the input of theregulator. It should be noted that harmonic injection in this context isnot limited to the traditional bass frequencies. It can be performed atall frequencies (with a fundamental frequency up to 12 KHz; after whichthe second harmonic is above the human hearing threshold);

6. Subharmonic Injection

When a signal is being limited in higher frequency bins by a regulator,compression feedback from the regulator can be provided to an embodimentof subsystem 102 (or 805). The subsystem can operate in response to thecompression feedback to generate subharmonics (having frequencies equalto (Fundamental frequency)/(n), where n is an integer) and to insert thesubharmonics into an audio input signal, thereby generating an enhancedsignal which is provided to the input of the regulator. This has theadvantage of working all the way up to 24 Khz. This will allow theperceived volume to increase when a user increases a volume control. Inother embodiments, the audio that is injected (to generate the enhancedsignal) is not a harmonic or subharmonic of a fundamental frequency(e.g., a “fundamental frequency” which is a frequency of a band ofsignificance, as indicated by compression feedback, or by feedback froma smart amplifier). For example, to enhance audio content for playbackby a bass speaker by injecting therein audio having a selected frequencywhich is below a fundamental frequency (where the fundamental frequencyis a frequency of a band of significance, as indicated by the feedback),the selected frequency may not be a subharmonic of the fundamentalfrequency, and may instead be determined based on the location of theresonant peak of the speaker (since the speaker may not be capable atthe subharmonic);

7. Virtualization

When performing virtualization (e.g., by operating an embodiment ofsubsystem 102 or 805), the virtualizer may operate in a mannercontrolled by compression feedback (e.g., from regulator 104) whichcompresses the output of the virtualizer. Virtualizers generally cause avolume change that might cause the regulator to limit certain bins. Insome cases, this would undesirably cause a collapse in spatial audio,unless operation of the virtualizer is controlled (in accordance with anembodiment of the invention) by the compression feedback.

An example of virtualization in response to compression feedback isdescribed with reference to FIG. 10. In another example ofvirtualization in response to compression feedback, the virtualizer doesnot virtualize a height filter in the case where it is causing (asindicated by the compression feedback) the regulator to limit a band,and instead just renders the audio to the listener plane. In anotherexample of virtualization in response to compression feedback, thevirtualizer reduces the amount of reverb (the “wet” component) withinthe signal and just keeps the anechoic feed (the “dry” component) whenthe regulator is limiting the relevant bins (as indicated by thecompression feedback); or

8. Equalization

When performing equalization (e.g., by operating an embodiment ofsubsystem 102 or 805), the equalizer may operate in a manner controlledby compression feedback (e.g., from regulator 104) which compresses theoutput of the equalizer. An equalizer preset may cause the regulator tostart limiting certain bins. The equalizer may decide (in response tothe compression feedback) to change to another preset to avoid thelimiting that occurs (as indicated by the compression feedback) due tothe regulator component.

In some embodiments; the inventive system (e.g., subsystem 102 of FIG. 3or the system of FIG. 3A) is configured to perform enhancement inresponse to compression feedback which indicates that compression isbeing applied in only in one band (e.g., the compression feedbackindicates that no (a zero amount of) compression is being applied ineach other band). For example, if regulator 104 is a multiband limiterwith the bands 1-1000 Hz and 1000-20000 Hz, and if the content consistsof a 500 Hz sine wave and this causes the speaker to distort, theregulator will not apply compression to the top band (1000-20000 Hz) andthe compression feedback will indicate this.

In some embodiments, the inventive system (e.g., subsystem 100 of FIG.3, or system 800 of FIG. 3A or a subsystem thereof) is configured toperform enhancement (in response to compression feedback) in the timedomain. For example, the enhancement may apply parametric filters (whichmay be implemented as time domain biquad filters). These parametricfilters may be used to implement an equalization bass enhancement. Foranother example, the enhancement may apply a parametric low pass filterthat adjusts its knee point based upon the compression feedback.

We next describe aspects of embodiments in which at least one ofdecoding, rendering, or audio enhancement is performed in response tofeedback from a smart amplifier (e.g., feedback 809 from smart amplifier808 of FIG. 3A). The feedback may be indicative of temperature, voltage,current, impedance, or resistance, of at least one speaker (e.g., atleast one coil of at least one speaker).

Smart amplifiers typically employ broadband compression for temperatureprotection. If they are employing broadband compression, the bassdistribution strategy may have two modes, one mode employing no bassdistribution and another mode that employs bass distribution with afixed cutoff (e.g., 400 Hz), with a sliding cutoff when transitioningbetween the two modes. If a smart amplifier is applying multibandcompression, temperature broadband limiting may be implemented byclipping the maximum excitation values per frequency bin (i.e., theequivalent of applying a broadband limit) and allowing low level signalsnot to be limited and distributed. The feedback employed in someembodiments of the invention may be indicative of the state of suchlimiting.

In some embodiments, audio may be rerouted (e.g., channels ofmultichannel audio may be routed to different elements of an audioenhancement subsystem) in response to feedback (e.g., temperaturefeedback) from a smart amplifier, before limiting occurs (i.e., limitingimplemented by the smart amplifier) to protect the speaker(s) and ensurethat the limiting is less likely to occur and that volume is maintained.

Smart amplifiers typically implement multiband compression for excursionprotection. A smart amplifier which implements multiband compression isconsidered (in the present disclosure) to be a multiband limiter, andfeedback generated by such a smart amplifier (and applied to control atleast one aspect of at least one of audio enhancement, decoding, orrendering in accordance with an embodiment of the present invention) isconsidered (in the present disclosure) to be compression feedback.

In some embodiments of the present invention, decoding, rendering,and/or audio enhancement is performed in response to compressionfeedback (e.g., feedback indicative of amount of compression applied orto be (e.g., predicted to be) applied to each of at least one frequencyband (e.g., each of at least two frequency bands) of a multichannelaudio signal, or thresholds of a multiband limiter, or power oramplitude of at least one channel of the multichannel audio signal, orstate of a system volume control), and/or feedback from a smartamplifier. For example, audio enhancement (e.g., including re-routing ofchannels of audio content) is performed in response to the feedback soas to prevent a multiband limiter from limiting or to improve audioquality when a multiband limiter is limiting.

The feedback from the smart amplifier (e.g., indicative of current(e.g., in cases in which it is desired to preserve battery life) ortemperature) may be employed to reduce the amount of energy that is sentto the speaker system in a way that prioritizes the important audiocontent. Use of such feedback (rather than compression feedbackindicative of amount of compression applied by a multiband limiter) maybe desirable to prevent the smart amplifier (or a multiband limiter)from applying compression. For example, if the feedback indicates thattemperature (of a speaker) is creeping up slowly, the decoding,rendering, and/or enhancement may be performed (in response to thefeedback) so as to reduce gradually reduce the energy that is being sentto the speaker before it is necessary to apply harsh compression.

With reference to FIG. 4, we next describe an example embodiment (system200) of the system of FIG. 3 or FIG. 3A. In system 200 of FIG. 4, bassvolume enhancer (bass extracting mixer) 201 is an implementation ofsubsystem 102 of FIG. 3, and the multichannel audio input signalprovided to enhancer 201 includes first channel 203 and second channel204. It should be appreciated that in some variations of the FIG. 4embodiment, the multichannel audio input provided to enhancer 201includes “n” channels (where “n” may be equal to 2, or not equal to 2).Each channel of the audio input may be a speaker channel or an audioobject.

In system 200, multiband limiter 202 and compression feedback signal 217(generated by limiter 202 and provided to enhancer 201) may beidentical, respectively, to limiter 104 and compression feedback signal105 of FIG. 3. In some implementations, limiter 202 may be configured togenerate feedback signal 217 such that feedback signal 217 is indicativeonly of an amount of compression applied by limiter 202 in at least onelow frequency band (or each of at least two low frequency bands), in alow frequency range (e.g., up to 500 Hz), of at least one channel (e.g.,one or both of channels 215 and 216) provided from enhancer 201 tolimiter 202. The audio output of limiter 202 (in response to channels215 and 216) is multichannel audio output signal 218. Typically, limiter202 operates on a sequence of blocks of audio values (provided theretoby enhancer 201), and feedback signal 217 is indicative (in a relevanttime window) of the gain(s) applied by limiter 202 to the audio value(s)in each different frequency band of the current block.

Bass volume enhancer 201 of FIG. 4 (and typical ones of the notedvariations thereof) generates (and provides to multiband limiter 202) abass-enhanced output audio channel (channel 215 or 216) in response toeach input audio channel (203 or 204). Bass volume enhancer 201 iscoupled and configured to vary the cutoff frequency of at least one bassextraction filter thereof (e.g., each of filters 205 and 206 of FIG. 4)in response to the amount of compression (as indicated by feedback 217)applied (in each of at least two frequency bands) by multiband limiter202, and to generate a bass-enhanced audio signal (e.g., an enhancedaudio signal including channels 215 and 216, as shown) in response tothe input signal channels (e.g., channels 203 and 204 as shown) providedthereto.

Typically, enhancer 201 generates (extracts) a low frequency audiosignal (e.g., each of signals 207 and 208 of FIG. 4) from each channel(or each of at least two channels) of the input audio signal providedthereto, sums together each extracted low frequency audio signal togenerate a summed signal (e.g., signal 214 of FIG. 4) which is amonophonic low frequency signal, and then mixes the summed signal back(at a particular level) into each of the channels from which a lowfrequency audio signal was extracted (e.g., mixes the summed signal withhigh frequency audio content of each channel of the input audio signalprovided to enhancer 201) to generate bass-enhanced output channels(e.g., output channels 215 and 216 of FIG. 4).

Still with reference to FIG. 4, it should be appreciated that manymodern laptops and soundbars (and other modern devices which outputsound or at least one audio signal) provide limited (e.g., weak) bassperformance. Such devices can be produced or modified to implement theFIG. 4 embodiment of the invention (or variations thereon), in order toincrease their output bass volume by dynamically routing low frequencyaudio content between channels. This trades spatial fidelity byspreading the bass content across multiple channels by downmixing it tomono (e.g., to monophonic signal 214 of FIG. 4) and then remixing itinto (i.e., mixing it with high frequency content of) the originalchannels. This is especially desirable in cases in which the bass (orother low frequency content) of input audio content has more energy insome channels (of the input audio content) than in other channels. Suchcases are common, e.g., where the input audio content is songs by “TheBeatles” or Dolby Atmos (or other object-based audio) content. In thiscontext, loss of spatial fidelity is typically acceptable because bassis less directional than high frequency content and users generallyvalue loudness (of low frequency content) over spatial fidelity.

Bass volume enhancer 201 of FIG. 4 is configured to change the routingof low frequency content of input audio provided thereto in response toamount of compression applied by multiband limiter 202 in order tohandle power characteristics of the speaker(s) of a playback (orrendering and playback) system to which the output of limiter 202 isprovided. An advantage of the FIG. 4 system is that multiband limiter202 makes the system stable, since any excessive (too aggressive) gainsapplied in element 209, 210, and/or 211 of FIG. 4 are limited (bylimiter 202) to be within what the relevant speaker(s) can handle.

The FIG. 4 system scales with the number of speakers. For example, avariation on enhancer 201 of FIG. 4 can include N branches (where N isan integer greater than 2), each branch including a bass extractor andmixing element, for generating N bass enhanced output signals (ratherthan two bass enhanced output signals, 215 and 216, as in FIG. 4) eachfor provision to a multiband limiter (e.g., limiter 202). In thisexample, mixing element 209 (of FIG. 4) is replaced by a mixing elementwhich sums together N low frequency signals (each of which is extractedin a different one of the branches) to generate a monophonic lowfrequency signal which is mixed with the high frequency contentextracted by each of the branches (or mixing element 209 is replaced bya mixing element which sums together only a subset of the N lowfrequency signals extracted in the branches, to generate the monophoniclow frequency signal which is mixed with the high frequency contentextracted by each of the branches). For another example, anothervariation on the FIG. 4 system can include M instances (where M is aninteger greater than 1) of system 200 of FIG. 4, each instance includinga bass volume enhancer (e.g., enhancer 201 or a variation thereon) and amultiband limiter, so that each instance of the bass volume enhancerenhances a different subset of a full set of input channels, and eachinstance of the limiter compresses the output of one of the bass volumeenhancers for playback (or rendering and playback) by a different subsetof a full set of speakers.

We next describe in more detail the implementation of enhancer 201 shownin FIG. 4. This implementation includes bass extraction filters 205 and206, and mixing elements 209, 211, and 212, coupled together as shown inFIG. 4. Each of bass extraction filters 205 and 206 has a cutofffrequency which is controlled by compression feedback 217.

Bass extraction filter 205 is configured to extract (from input channel204) high frequency content (signal 213) in a high frequency range abovethe cutoff frequency, and low frequency content (signal 208) in a lowfrequency range below the cutoff frequency. Bass extraction filter 206is configured to extract (from input channel 203) high frequency content(signal 212) in a high frequency range above the cutoff frequency, andlow frequency content (signal 207) in a low frequency range below thecutoff frequency. The cutoff frequency is controlled by the amount oflimiting (as indicated by feedback 217) that is being performed inlimiter 202 (typically, only in a low range of frequencies, for exampleup to 500 Hz, but alternatively in a wider or full range offrequencies). Typically, compression feedback 217 is indicative of (andthe cutoff frequency of each of filters 205 and 206 is determined by) anamount of compression applied by limiter 202 in each of at least two lowfrequency bands of a low range of frequencies (e.g., up to 500 Hz).Alternatively, compression feedback 217 is indicative of (and the cutofffrequency of each of filters 205 and 206 is determined by) an amount ofcompression applied by limiter 202 in frequency bands in a wider or fullrange of frequencies.

Mixing element 209 is coupled and configured to sum together (e.g.,generate a weighted sum of) the low frequency signals 207 and 208extracted by filters 205 and 206, to generate a monophonic low frequencysignal (channel) 214. Signal 214 is provided to each of mixing elements210 and 211. High frequency signal 212 (extracted by filter 206) is alsoprovided to mixing element 211, and element 211 sums together signals214 and 212 to generate bass-enhanced output audio channel 216. Highfrequency signal 213 (extracted by filter 205) is provided to mixingelement 210, and element 210 sums together signals 214 and 213 togenerate bass-enhanced output audio channel 215.

In a typical implementation, each of filters 205 and 206 is configuredto determine the cutoff frequency (“targeted_cutoff” in the formulasbelow) in accordance with the following formulas:

targeted_cutoff=total_gain_ratio*max_freq_limiting*aggressiveness,

where “aggressiveness” is a parameter indicative of aggressiveness ofbass volume enhancement (this parameter may be tuned by ear, e.g., bythe user or algorithm creator, to ensure that the system does notinclude too much or too little energy in monophonic low frequency signal(channel) 214),

“max_freq_limiting” is the maximum frequency covered by a band that sbeing limited in the regulator (it is typically determined by, orderived directly from, the highest frequency of the highest-frequencyband that is being limited by the regulator. In some implementations, itis clipped to the range that the bass extraction filter supports), and

“total_gain_ratio”=total_gain/max_possible_gain, where

“max_possible_gain” is the sum of the maximum gains of every band thatis being limited (at the time) by limiter 202 for all bands that may bebass extracted in element 205 or 206 (or all bands that may be limitedin limiter 202, in some embodiments). It is the sum of the maximum gainsthat may be applied (by limiter 202) for all bands that may be bassextracted (in element 205 or 206) in the sense that it is the maximumintegral of all the gains that may be applied (by limiter 202) for thebins having frequency that do not exceed a maximum cutoff frequency ofelement 205 or 206, and

“total_gain” is the sum of all gains being applied (as indicated by thefeedback 217 from multiband limiter 202 for each frequency band) to allbands that may be bass extracted (or all bands that may be limited, insome embodiments).

The “total_gain_ratio” is an indicator of how much the regulator(limiter 202) is limiting overall within all the bands that may be bassextracted (in element 205 or 206). It is normalized (by the“max_possible_gain” parameter) so that it gives a better indication ofthe overall amount of limiting that is occurring for a variable numberof bands.

The cutoff frequency (“targeted_cutoff” in the above formula) for eachof filters 205 and 206 is increased, to increase the amount of bassenhancement applied, when the limiter is applying more limiting (e.g.,when “total_gain_ratio” in the formulas is increased). The cutofffrequency for each of filters 205 and 206 is decreased, to decrease theamount of bass enhancement applied, when the limiter is applying lesslimiting (e.g., when “total_gain_ratio” in the formulas is decreased).

The cutoff frequency (i.e., “targeted_cutoff” in the above formulas) ispreferably smoothed out with an attack and release to ensure the userdoes not notice sudden jumps in the panning.

If limiter 202 is not limiting (compressing), the cutoff frequencybecomes so low that both low frequency content signals (207 and 208) aresilent or substantially silent. As compression by limiter 202 increases(e.g., in one or more frequency bands, or across more bands), the cutofffrequency (of each of filters 205 and 206) increases so that the energyof each of signals 207 and 208 typically increases (and the energy ofeach of signals 212 and 213 typically decreases correspondingly), untilthe compression increases to a threshold amount at (and above) which thecutoff frequency (of each of filters 205 and 206) reaches its maximumvalue (e.g., 500 Hz). Thus, as limiter 202 applies more compression, themonophonic low frequency signal 214 is boosted, causing audio outputsignal 218 of limiter 202 to be indicative of more monophonic (“mono”)bass. As limiter 202 applies less compression, the energy of monophoniclow frequency signal 214 is reduced (causing the audio output signal 218of limiter 202 to be indicative of less mono bass). Signal 218 can thusprovide increased bass performance (relative to that which input signalchannels 203 and 204 could themselves provide, if provided directly tolimiter 202), without causing the relevant speaker(s) to distort.

In some implementations, the maximum cutoff frequency for filters 205and 206 is less than 500 Hz (to exclude more, or all, midrange contentfrom signal 214) or greater than 500 Hz.

The system of FIG. 4 can be further enhanced by adding a virtual basssubsystem (e.g., a stage or other subsystem which applies virtual bass)that is also controlled by compression feedback from the multibandlimiter. For example, FIG. 5 is a block diagram of an embodiment (system300) of the inventive system which includes the system of FIG. 4, andalso a virtual bass stage comprising elements 219 and 220. Otherembodiments of the invention implement only virtual bass enhancement(e.g., as implemented by one or both of elements 219 and 220) inresponse to compression feedback (e.g., feedback 217 from limiter 202),but not bass volume enhancement of the type implemented in the FIG. 5embodiment by subsystem 201.

In system 300 of FIG. 5, the virtual bass stage (implemented by elements219 and 220) is coupled and configured to apply virtual bass to each ofchannels 215 and 216 (which are output from subsystem 201), to generateenhanced channels 221 and 222 which are provided to limiter 202.Alternatively, the virtual bass stage (219 and 220) can be coupled andconfigured to apply virtual bass to each of input channels 203 and 204to generate enhanced channels, these enhanced channels are provided toelements 205 and 206 of subsystem 201, so that the audio content of theoutput channels (215 and 216) of subsystem 201 have undergone bothvirtual bass enhancement (in response to compression feedback fromlimiter 202) followed by enhancement in subsystem 201 (also in responseto compression feedback from limiter 202).

In the FIG. 5 system, each of virtual bass elements 219 and 220 isconfigured to add harmonics (of a missing fundamental) to the channel(channel 216 or 215) it receives when the missing fundamental (a“missing” fundamental frequency) cannot be reproduced or reproduced well(relying on the well-known missing fundamental phenomenon). Thecompression feedback 217 from limiter 202 is indicative of whether sucha fundamental (in a frequency band in which limiter 202 can applycompression) cannot be reproduced or reproduced well. Thus, multibandlimiter 202 provides feedback to each of elements 219 and 220 which isindicative of whether each relevant fundamental can be reproduced (orreproduced well). In a typical implementation, each of virtual basselements 219 and 220 adds only one harmonic in response to anon-reproducible (and thus “missing”) fundamental, since adding more istypically too aggressive and sounds artificial.

In the example implementation described below, each of elements 219 and220 inspects the first (lowest) band (in the sense that it responds tothe lowest frequency band indicated by feedback 217) to determinewhether it includes a missing fundamental frequency. The exampleimplementation set forth below, element 219 (or 220) determines theamount of virtual bass (“virtual bass gains”) it applies, in response tocompression feedback 217 from limiter 202, as indicated by the followingexponential formula:

virtual_bass_gains=min_virtual_bass_gain+((1+0.01×A)^(−regulator_gain)−1)

where

“regulator_gain” is the regulator gain value (fed back from multibandlimiter 202 to the relevant one of element 219 or 220) for the lowestfrequency band;

“A” is a parameter indicative of aggressiveness of virtual bassapplication (i.e., how much virtual bass is applied per amount ofregulator gain), e.g., A=−25; and

“min_virtual_bass_gain” is the minimum amount of virtual bass gainapplicable.

Alternative methods (e.g., log and linear formulas, in contrast with anexponential formula as set forth above) may be implemented to determinethe amount of virtual bass applied (e.g., by element 219 or 220) inresponse to the compression feedback from the multiband limiter. In atleast some applications, an exponential formula (e.g., as set forthabove) is superior since it prevents application of significant virtualbass until it is necessary to do so.

In FIG. 5, the output channel of virtual bass element 219 (channel 221)and the output channel of virtual bass element 220 (channel 222) shouldhave no harmonics (virtual bass) added therein when multiband limiter202 is not limiting in the lowest band. When limiter 202 is limiting inthe lowest band (as indicated by feedback 217), then elements 219 and220 should add harmonics into channels 221 and 222 (because thefundamental will be missing, or poorly reproduced, from the output ofthe speaker(s) which play back the relevant content).

It is often desirable to implement both virtual bass enhancement (e.g.,by operation of elements 219 and 220) and bass distribution (e.g., byoperation of subsystem 201) together since they typically act atdifferent times within a program (e.g., a song) and complement eachother.

With reference to FIG. 6, we next describe an embodiment of theinventive system which is useful in connection with playback of audio bymultispeaker devices (especially those including speakers of differentclasses), and which employs more than one multiband limiter (i.e., amultiband limiter per speaker class) to implement dynamic crossovers.

In a commercially important class of multispeaker devices, each suchdevice includes (or is used to drive) at least two speakers having verydifferent characteristics. Traditionally, the playback of audio by sucha device has been protected using a global multiband limiter to preservespatial integrity. Using multiple limiters in accordance with a class ofembodiments of the invention (e.g., the FIG. 6 embodiment), in which thelimiters include one limiter per class of speaker, can improve the powerhandling of (and typically also the audio playback implemented using)such a device.

FIG. 6 is a block diagram of a system (system 400) configured to performaudio enhancement and compression (on a per speaker class basis) inaccordance with an embodiment of the invention. System 400 is configuredto perform audio enhancement in the sense of filtering and routing theaudio content according to the speaker handling characteristics at thetime of playback. This routing is acceptable since the bass contentgenerally is not very directional when played. System 400 includesmultiband limiters (406, 407, and 408) which are employed on a perspeaker class basis in the sense that:

limiter 406 performs multiband compression on a signal (signal 403) togenerate an output signal (having one or more channels) for playback (orrendering and then playback) by one or more tweeters (e.g., of amultispeaker device);

limiter 407 performs multiband compression on a signal (signal 404) togenerate an output signal (having one or more channels) for playback (orrendering and then playback) by one or more mid-range speakers (e.g., ofthe same multispeaker device); and

limiter 408 performs multiband compression on a signal (signal 405) togenerate an output signal (having one or more channels) for playback (orrendering and then playback) by one or more low-frequency range speakers(e.g., of the same multispeaker device).

In typical implementations, each of limiters 406 and 407 providescompression feedback that controls the content and routing of sets ofchannels to all the limiters (406, 407, and 408) on a frequencydependent basis using crossovers. In typical ones of theseimplementations, compression feedback 409 from limiter 406 is indicativeof compression applied by limiter 406 in each of at least two frequencybands of each of channel(s) 403, and compression feedback 410 fromlimiter 407 is indicative of compression applied by limiter 407 in eachof at least two frequency bands of each of channel(s) 404. Typically,each of feedback 409 and feedback 410 is indicative of regulator gains(applied by the relevant one of limiters 406 and 407), but alternativelyeach can be indicative of compression feedback of another type. In somealternative implementations, each of feedback 409 and feedback 410 isreplaced by (or includes) feedback other than compression feedback(e.g., feedback indicative of amount of harmonic and/or intermodulationdistortion predicted for the relevant driver(s), or other feedbackresulting from a mechanism other than gain application by a multibandlimiter).

For example, in some implementations, feedback 409 is replaced by (orindicative of) feedback indicative of either the power or amplitude ofsignal 403 entering limiter 406 within at least one band, and feedback410 is replaced by feedback indicative of either the power or amplitudeof signal 404 entering limiter 407 within at least one band. This allowsthe cutoff frequency of each of elements 411 and 412 to start fading(changing) before the speaker(s) start distorting. More generally,feedback 409 and/or 410 may be indicative of power or amplitude of atleast one of the signal(s) to be compressed by at least one of multibandlimiters 406, 407, or 408). Reasons for employing feedback of the typesnoted in this paragraph include: operation of the system in accordancewith a psychoacoustic model in which perceived loudness is different atdifferent volume levels for different frequencies; and/or preservingeach speaker over a longer term (in some cases, use of compressionfeedback indicative of regulator gains may allow some distortion throughto the speakers); and/or increasing electrical efficiency.

For another example, in some implementations, feedback 409 and 410 arereplaced by (or indicative of) feedback indicative of state of a systemvolume control (which may be indicative of power or amplitude of atleast one of the signal(s) to be compressed by at least one of multibandlimiters 406, 407, or 408).

For another example, in some implementations, feedback 409 and 410 arereplaced by (or indicative of) feedback indicative of amount and type ofdistortion that is being or is to be blocked (by operation of themultiband limiters). Such feedback is an example of compression feedbackindicative of amount of compression to be (e.g., predicted to be)applied by limiter 104. For example, 5% total harmonic distortion and 4%intermodulation distortion may be occurring due to the signals within aparticular band. If intermodulation distortion is high then the system'sprocessing may be applied differently (e.g., to generate each set ofchannels 403, 404, and 405 to have different content and/or number ofchannels) than if harmonic distortion is at a similar percentage (sinceintermodular distortion typically sounds much worse).

The input audio signal (signal 401) provided to the FIG. 6 system is amultichannel audio signal including a set of “n1” channels (which aretypically full bandwidth channels).

The FIG. 6 system also includes:

crossover and gain stage 411 which is configured to split and filter(high-pass filter and low-pass filter) multichannel input audio signal401 (including “n1” channels) into high frequency channel(s) 403(comprising “n2” channels of high frequency components) and lowfrequency channels(s) 402 (comprising “n3” channels of low frequencycomponents) in response to compression feedback 409 from limiter 406,where n1, n2, and n3 are integers. Compression feedback 409 providedfrom limiter 406 to stage 411 is preferably smoothed (with differentattack and release times) to ensure the dynamic nature of the system ismasked from the user; and

crossover and gain stage 412 which is configured to split and filter(high-pass filter and low-pass filter) low frequency channel(s) 402 intorelatively high frequency channel(s) 404 (comprising “n4” channels ofhigh frequency components) and relatively low frequency channels(s) 405(comprising “n5” channels of low frequency components) in response tocompression feedback 410 from limiter 407, where n4 and n5 are integers.Compression feedback 410 provided from limiter 407 to stage 412 ispreferably smoothed (with different attack and release times) to ensurethe dynamic nature of the system is masked from the user.

Stage 411 has a first crossover frequency (controlled in response tocompression feedback 409), and channel(s) 403 include high-pass filteredcontent (of signal 401) having frequency above the first crossoverfrequency, as filtered by high pass filter(s) of stage 411. Channel(s)402 include low-pass filtered content (of signal 401) having frequencybelow the first crossover frequency, as filtered by low pass filter(s)of stage 411. The gains applied by stage 411 to channels 401 (togenerate channels 402 and 403) are also controlled in response to thecompression feedback 409.

Stage 412 has a second crossover frequency (controlled in response tocompression feedback 410), and channel(s) 404 include high-pass filteredcontent (of channel(s) 402) having frequency above the second crossoverfrequency, as filtered by high pass filter(s) of stage 412. Channel(s)405 include low-pass filtered content (of channel(s) 402) havingfrequency below the second crossover frequency, as filtered by low passfilter(s) of stage 412. The gains applied by stage 412 to channel(s) 402(to generate channels 404 and 405) are also controlled in response tothe compression feedback 410.

Multiband limiter 406 for the tweeter class of speakers limits the “n2”channels 403 and thus speaker feeds corresponding thereto. Each ofchannels 403 may be a speaker feed (in which case there are “n2”tweeters), or speaker feeds can be generated in response to channels403.

Multiband limiter 407 for the mid-range class of speakers limits the“n4” channels 404 and thus speaker feeds corresponding thereto. Each ofchannels 404 may be a speaker feed (in which case there are “n4”mid-range speakers), or speaker feeds can be generated in response tochannels 404.

Multiband limiter 408 for the low-range class of speakers limits the“n5” channels 405 and thus speaker feeds corresponding thereto. Each ofchannels 405 may be a speaker feed (in which case there are “n5”low-range speakers), or speaker feeds can be generated in response tochannels 405.

The FIG. 6 system scales with the number of speaker types (classes). Forexample, where there are “X” classes of speakers (where X is someinteger), a scaled version of the FIG. 6 system may include “X” limitersand “X−1” (e.g., a cascade of “X−1”) crossover and gain stages. Althoughthe FIG. 6 system includes a cascade of two crossover and gain stages(411 and 412), in variations on the FIG. 6 embodiment (or scaledversions thereof), the crossover and gain stages may be implemented inparallel (rather than as a cascade) to reduce the latency of thesystem's response to limiting.

System 400 of FIG. 6 (or variations thereon) can be further extended byimplementing bass volume enhancement (e.g., of a type implemented insystem 200 of FIG. 4) and/or virtual bass enhancement (e.g., of a typeimplemented in elements 219 and 220 of system 300 of FIG. 5) and/oranother type of audio enhancement. System 500 of FIG. 7 is an example ofsuch a combined system.

System 500 is configured to perform audio enhancement including bassenhancement (and filtering and routing of the audio content according tothe speaker handling characteristics at the time of playback). In system500, bass content is distributed (e.g., as described with reference tosystem 200) to different speakers within each class, and/or virtual bassenhancement is performed (e.g., as described with reference to system300), in response to compression feedback from each multiband limiter(i.e., each of limiters 406, 407, and 408).

Elements of system 500 of FIG. 7 that are identical to correspondingelements of system 400 (of FIG. 6) are numbered identically in FIGS. 6and 7.

Each of elements 413, 414, and 415 of system 500 of FIG. 7 is configuredto perform bass volume enhancement of the type implemented by subsystem201 of FIG. 4, and/or virtual bass enhancement of the type implementedby element 219 (or 220) of FIG. 5.

System 500 of FIG. 7 implements a combination of bass distribution(and/or virtual bass enhancement) on a per regulator basis, and routesaudio content to each class of speakers in accordance with the speakercharacteristics of each class, across a system that has multipleregulators (406, 407, and 408). Each of the regulators is for adifferent speaker class.

Element 413 of FIG. 7 is configured to implement bass distribution ofthe type performed by subsystem 201 of FIG. 4; and/or virtual bassenhancement of the type performed by one or both of elements 219 and 220of FIG. 5. For example:

element 413 (together with limiter 406) may implement the system of FIG.5 (or a variation thereon), with limiter 406 corresponding to limiter202 of FIG. 5, and with enhancement implemented by element 413 inresponse to compression feedback 409 from limiter 406. Preferably,element 413 implements smoothing of the type described above (e.g.,smoothing of feedback signal 409);

element 414 (together with limiter 407) may also implement the system ofFIG. 5 (or a variation thereon), with limiter 407 corresponding tolimiter 202 of FIG. 5, and with enhancement implemented by element 414in response to compression feedback 410 from limiter 407. Preferably,element 414 implements smoothing of the type described above (e.g.,smoothing of feedback signal 410); and

element 415 (together with limiter 408) may also implement the system ofFIG. 5 (or a variation thereon), with limiter 408 corresponding tolimiter 202 of FIG. 5, and with enhancement implemented by element 415in response to compression feedback 419 from limiter 408. Preferably,element 415 implements smoothing of the type described above (e.g.,smoothing of feedback signal 419).

The enhanced output 416 of element 413 may include bass distributedtweeter channels consisting of high frequency components 403 of theoutput of element 411 with virtual bass applied thereto. The enhancedoutput 416 is provided to limiter 406. The enhanced output 417 ofelement 414 may include bass distributed midrange channels consisting ofhigh frequency components 404 of the output of element 412 with virtualbass applied thereto. The enhanced output 417 is provided to limiter407. The enhanced output 418 of element 415 may include bass distributedlow range channels consisting of low frequency components 405 of theoutput of element 412 with virtual bass applied thereto. The enhancedoutput 418 is provided to limiter 408.

With reference to FIG. 8, we next describe a class of embodiments of theinventive system which implement multichannel dialog enhancement inresponse to compression feedback. System 600 of FIG. 8 is an example ofsuch a system. Embodiments in this class implement channel-based dialogenhancement to increase dialog intelligibility, typically by duckingnon-center channels of a multichannel audio input when the compressionfeedback indicates that the regulator gains are limiting in the dialogbands. Optionally also, frequency dialog enhancement is performed on thecenter channel of the multichannel audio input in response to thecompression feedback. In variations on the embodiment shown in FIG. 8,all or some of the input channels are object channels (audio objects),the non-dialog objects are ducked, and dialog-enhancing gain can beapplied to the dialog objects (dialog objects can either be extractedfrom normal objects, or they can be labeled as dialog objects within abitstream).

The frequency bands in which dialog (human utterances) typically occurs,where intelligibility of the dialog will typically be significantlyincreased if content in such bands is enhanced, will be referred toherein as dialog bands.

In system 600 of FIG. 8, the multichannel audio input signal includesleft channel 601, right channel 602, center channel 603, left surroundchannel 604, and right surround channel 605. Gain element 606 is coupledand configured to apply a negative gain (which may be frequencydependent) to channel 601 in response to compression feedback 617 frommultiband limiter 616 (when the feedback 617 indicates that gainsapplied by limiter 616 in the dialog bands are limiting). Gain element607 is coupled and configured to apply a negative gain (which may befrequency dependent) to channel 602 in response to compression feedback617 from multiband limiter 616 (when the feedback 617 indicates thatgains applied by limiter 616 in the dialog bands are limiting). Gainelement 609 is coupled and configured to apply a negative gain (whichmay be frequency dependent) to channel 604 in response to compressionfeedback 617 from multiband limiter 616 (when the feedback 617 indicatesthat gains applied by limiter 616 in the dialog bands are limiting).Gain element 610 is coupled and configured to apply a negative gain(which may be frequency dependent) to channel 605 in response tocompression feedback 617 from multiband limiter 616 (when the feedback617 indicates that gains applied by limiter 616 in the dialog bands arelimiting).

In a typical implementation, gain element 608 is coupled and configuredto apply to center channel 603 a dialog-enhancing gain (which may befrequency dependent, in the sense that it may consist of different gainsapplied in different frequency bands of center channel 603) in responseto compression feedback 617 from multiband limiter 616. For example,element 608 may apply unity gain (so that channel 613 is identical tochannel 603) when feedback 617 indicates that that gains applied bylimiter 616 in the dialog bands are not limiting (or not significantlylimiting), and element 608 may apply frequency-dependent gain to channel603 (e.g., to emphasize content in non-dialog bands relative to contentin dialog bands) in response to feedback 617 indicating that that gainapplied by limiter 616 in at least one of the dialog bands is limiting.

In alternative implementations, gain element 608 is omitted (or alwaysapplies unity gain), so that channel 613 is always identical to channel603.

The output of element 606 is ducked channel 611. It is identical tochannel 601 when feedback 617 indicates that that gains applied bylimiter 616 in the dialog bands are not limiting (or not significantlylimiting). The output of element 607 is ducked channel 612. It isidentical to channel 602 when feedback 617 indicates that that gainsapplied by limiter 616 in the dialog bands are not limiting (or notsignificantly limiting). The output of element 609 is ducked channel614. It is identical to channel 604 in response to feedback 617indicating that that gains applied by limiter 616 in the dialog bandsare not limiting (or not significantly limiting). The output of element610 is ducked channel 615. It is identical to channel 605 in response tofeedback 617 indicating that that gains applied by limiter 616 in thedialog bands are not limiting (or not significantly limiting).

The output of element 608 is dialog enhanced channel 613. Channel 613 istypically identical to channel 603 when feedback 617 indicates that thatgains applied by limiter 616 in the dialog bands are not limiting (ornot significantly limiting). In response to feedback 617 indicating thatthat gains applied by limiter 616 in the dialog bands are limiting,element 608 may apply a dialog-enhancing gain (which may befrequency-dependent gain) to channel 603 to generate channel 613. Inthis latter case, the dialog-enhancing gain applied by element 608 mayboost dialog bands of channel 603 differently than other bands ofchannel 603. A downmixer (not shown in FIG. 8) may exist between theoutputs of dialog enhancer subsystem (subsystem 618 in FIG. 8) and theinputs of the multiband limiter (multiband limiter 616 in FIG. 8).

Compression feedback 617 from multiband limiter 616 may be indicative ofgains applied in all bands of all or some of channels 611, 612, 613,614, and 615 (e.g., only channel 613). Alternatively, compressionfeedback 617 may be indicative of gains applied only in dialog bands ofall or some of channels 611, 612, 613, 614, and 615 (e.g., only channel613).

In some implementations, feedback 617 can be replaced by (or include)feedback other than compression feedback (e.g., it may be feedback froma smart amplifier). In some implementations the feedback may beindicative of state of a system volume control, as is the feedback insome above-noted implementations of FIG. 6. Such volume control feedbackmay be useful since it may indicate that a user is informing the signalchain that he or she cannot understand the dialog emitted from thespeaker(s) (in the same or similar way that compression feedbackindicative of gains applied by limiter 616 may indicate, by allowing aninference, that a user cannot understand the dialog), causing thesystem's dialog enhancement processing to change accordingly.

Subsystem 618 of system 600 of FIG. 8 (which includes elements 606, 607,608, 609, and 610) is a ducking based dialog enhancer subsystem.

In some variations on the FIG. 8 system, multiband limiter 616 isreplaced by two or more multiband limiters (e.g., one for each differentclass of speakers, as are, for example, limiters 406, 407, and 408 ofFIG. 7). One of these limiters (to be referred to as the center channellimiter) is coupled and configured to apply compression to centerchannel 613. In such variations, subsystem 618 may be implemented in anyof the ways described above with reference to FIG. 8, except that thefeedback for each channel (i.e., the feedback applied to each ofelements 606, 607, 608, 609, and 610) is generated in the center channellimiter (i.e., it is compression feedback indicative of an amount oflimiting by the center channel limiter). In some such variations,multiple limiters are used even though all speakers belong to the sameclass, to ensure that the center channel is always intelligible.

In order to provide the user a more compelling experience, someembodiments of the inventive system implement a combination of channelbased dialog enhancement (e.g., as described with reference to FIG. 8),frequency dependent dialog enhancement, and broadband limiting (i.e.,timbre preservation). FIG. 9 is a block diagram of a system (system 620)which implements a combination of those three algorithmic components.

In system 620 of FIG. 9:

multichannel audio input signal 621 is provided to stage 622. Signal 621may be a 5.1 channel input audio signal, or another multichannel signal(typically having more than two channels) from which clean dialog can beextracted;

stage 622 is a frequency based dialog enhancement subsystem, configuredto apply a filter that emphasizes frequencies in the dialog bands, inresponse to compression feedback 617 from multiband limiter 625 (orother feedback, e.g., of any of the types described above), to generateoutput signal 623. The filter may perform filtering on clean dialog ordialog mixed into the channels of signal 621. Typically, any adjustmenton the amount of dialog enhancement implemented by the filter issmoothed. If multiband limiter 625 is limiting in the dialog bands (asindicated by compression feedback), then the dialog bands are emphasizedby subsystem 622, so that signal 623 is a set of channels indicative ofdialog which has been enhanced by frequency based dialog enhancementsubsystem 622;

stage 618 is a ducking based dialog enhancer subsystem (e.g., oneidentical to subsystem 618 of FIG. 8) which applies channel based dialogenhancement processing in response to compression feedback 617 frommultiband limiter 625 (or other feedback, e.g., of any of the typesdescribed above), to generate output signal 624 in response to signal623. Subsystem 618 implements channel based dialog enhancement (i.e.ducking of non-center (e.g., surround) channels with possible boostingof the center channel); and

stage 625 is a multiband limiter which is configured specifically toimplement timbre preservation in the dialog bands (and bands adjacentthereto), and which generates an output signal (for playback orrendering and playback) in response to signal 624. Limiter 625implements wider band limiting in particular bands (i.e., the dialogbands and the bands adjacent thereto) to preserve timbre of audiocontent. When limiter 625 limits within the dialog bands, the adjacentbands are also turned down (limited), and feedback 617 is generated soas to cause the frequency dependent dialog enhancement (performed bysubsystem 622) to be increased (turned up), thereby increasing theintelligibility of dialog within the limits of the speaker system.

In variations on system 620 of FIG. 9, the order of frequency dependentdialog enhancement subsystem 622 and channel based dialog enhancementsubsystem 618 is reversed, so that the channel based dialog enhancementsubsystem (e.g., which performs the functions of subsystem 618) operateson input signal 621, the frequency dependent dialog enhancementsubsystem (e.g., which performs the functions of subsystem 622) operateson the output of subsystem 618, and limiter 625 operates on the outputof the frequency dependent dialog enhancement subsystem.

With reference to FIG. 10, we next describe a class of embodiments ofthe inventive system which implement devirtualization in response tocompression feedback, to control (e.g., lower) an amount ofvirtualization which would otherwise be applied. In some suchembodiments, the compression feedback is indicative of state of a systemvolume control (in some cases, implementing such embodiments may requirespecial tuning by field engineers).

Some virtualizers result in a volume and/or intelligibility reductionwhen virtualizing.

By reducing the virtualization when a multiband limiter is activelylimiting, the loudness and intelligibility of an audio program (e.g., amovie soundtrack) can be improved, possibly at the cost of spatialfidelity.

Thus, some embodiments of the invention use a virtualizer start band asa control mechanism (responsive to compression feedback or otherfeedback) for the amount of virtualization that occurs. On low frequencybands (below the start band), virtualization is not applied. On highfrequency bands (above the start hand) virtualization is applied.Preferably, changes of the virtualizer start band (in response to thecompression feedback or other feedback) are heavily smoothed to ensurethat the user does not experience spatial collapse. Alternatively, theamount of virtualization performed individual frequency bands iscontrolled in response to compression feedback (or other feedback),e.g., from a multiband limiter.

In some alternative implementations, rather than (or in addition to)adjusting the start frequency for virtualization in response tofeedback, the speaker angle of a virtualizer is controlled in responseto compression feedback from a multiband limiter (or other feedback), toprovide a reduction in the virtualization effect as the angle approacheszero.

In the system (virtualizer 700) of FIG. 10:

virtualizer stage 702 is configured to receive multichannel audio inputsignal 701 provided thereto. Signal 701 may be a 5.1 channel input audiosignal, or another multichannel signal (typically having more than twochannels);

virtualizer stage (subsystem) 702 is configured to implement anadjustment control which adjusts the amount (or another characteristic)of virtualization implemented by subsystem 702 in one or more bands (atleast in one or more low bands) in response to feedback (e.g.,compression feedback 705 from multiband limiter 704 indicative ofwhether the band(s) are being limited). Typically, if limiter 704 islimiting in the lowest “n” bands then virtualization will not beperformed on those “n” bands. Preferably, subsystem 702 implements along smoother (to smooth changes in amount(s) or other characteristic(s)of virtualization in response to feedback), to reduce perceptibility ofany spatial collapse that results from changes in the virtualization inresponse to the feedback. In an alternative implementation, subsystem702 adjusts the angle of the virtualizer in response to compressionfeedback 705 from multiband limiter 704 (preferably with smoothing),thus implementing a broadband gradual reduction of the virtualization inresponse to more limiting by limiter 704. The virtualization implementedby subsystem 702 can either be a frequency magnitude basedvirtualization only (e.g., as in a height filter) or a more complicatedvirtualization (e.g., a speaker virtualizer as implemented within theDolby Audio API);

multiband limiter 704 is coupled and configured to perform limiting onthe output signal 703 of virtualizer subsystem 702, and to generate (andprovide to subsystem 702) compression feedback 705. As noted, in someembodiments subsystem 702 is configured to perform virtualization inresponse to feedback other than compression feedback from limiter 704,and in such embodiments limiter 704 need not generate compressionfeedback; and

virtualized signal 703 (e.g., a stereo signal, when signal 701 is a 5.1channel audio signal) is generated in subsystem 702 and provided tomultiband limiter 704. In some implementations in which signal 703includes more than two channels, subsystem 702 may change the routing ofvirtualized audio content between channels of signal 703 in response tocompression feedback 705 (e.g., as audio content is routed betweenchannels as described above with reference to the FIG. 4 embodiment)thus creating a different virtualized experience.

We next describe other embodiments of the inventive system which performaudio processing in response to compression feedback from a multibandlimiter (or in response to other feedback).

In some such embodiments, object-based audio rendering is performed inresponse to compression feedback. Conventional rendering of object-basedaudio has generally ignored the speaker characteristics of the speakersystem which is to implement playback. In accordance with someembodiments of the invention, object-based audio rendering (e.g., a typeor characteristic thereof) is changed in response to compressionfeedback from a multiband limiter.

For example, with reference to FIG. 3A, signal 801 of FIG. 3A mayinclude channels (object channels) indicative of audio objects. In casesin which they do, rendering subsystem 803 may be an object-based audiorendering system configured to prioritize audio objects (indicated bysignal 801 and the decoded version of signal 801 provided to subsystem803) based upon a priority field of each object (also indicated bysignal 801, or metadata corresponding to each audio object of signal801) such that when multiband limiter (regulator) 806 limits audiocontent of at least one audio object (as indicated by feedback 807provided by (e.g., generated and provided by) limiter 806 to subsystem803), or when feedback 809 indicates that low priority objects shouldnot be rendered, low priority objects are not rendered by subsystem 803whereas high priority objects are rendered normally by subsystem 803. Insuch implementations of FIG. 3A, feedback 807 may also be provided fromlimiter 806 (and/or feedback 809 may be provided from amplifier 808) tosubsystem 805 to control operation of subsystem 805 (e.g., as in anyembodiment of 805, or subsystem 102 of FIG. 3, described herein), but insome cases feedback 807 is not provided from limiter 806 to subsystem805.

Another example of a feedback mechanism (in which feedback 807 and/or809 of FIG. 3A is provided to rendering subsystem 803 to controloperation of subsystem 803) is performed in any of the following cases:signal 801 includes two sets of object streams (object channels); orsignal 801 includes one set of speaker channels and one set of objectstreams; or signal 801 includes two sets of speaker channels. In thisexample, rendering by subsystem 803 includes mixing of content of suchchannels (speaker and/or object channels) in response to feedback 807and/or 809 (e.g., with a priority of each individual channel that is ormay be mixed determined by feedback 807 and/or 809, e.g., so that thechannels have different relative priorities when regulator 806 islimiting than when it is not limiting). This is preferably implementedto ensure that the user will always be able to hear the most importantparts of the content.

The type of rendering by subsystem 803 may change in response tofeedback 807 and/or 809 (and/or in response to other feedback, e.g.,compression feedback, provided to subsystem 803). For example, considera feature of object-based audio rendering (OAR) known as “channeldistribution” (which is different than bass distribution). Channeldistribution spreads content of speaker channels (being rendered withobject channels) to nearby speakers according to a gain curve. In someembodiments of the inventive system, channel distribution in OAR (e.g.,implemented by subsystem 803 of FIG. 3A) is controlled (e.g., turned onor modified) for all speaker channels when regulator 806 starts limiting(as indicated by feedback 807), e.g., to spread the energy of thechannels to more speakers. For example, subsystem 803 may change agradient of the gain curve of the channel distribution in response tofeedback 807 and/or 809, so that the gain curve becomes shallower, thusincreasing the amount of energy pumped (as a result of rendering bysubsystem 803) into at least some of the speakers of a speaker system(e.g., speaker system 109) as regulator 806 limits.

With reference to FIG. 11, an example of such channel distribution willbe described. FIG. 11 assumes that the signal (e.g., the decoded signaloutput from subsystem 802 of FIG. 3A) is to be rendered (e.g., bysubsystem 803 for playback by a speaker system) to at least eightspeaker channels (channels Ls1, Ls, Ls2, Lrs1, Lrs, Lrs2, Lcs, and Cs),each corresponding to a different speaker of the speaker system (e.g.,speaker system 109). The content to undergo channel distribution may beindicated by one speaker channel (channel “Ls,” in FIG. 11) or oneobject channel, and the channel distribution may be implemented bycopying the content to the other channels with an adjustable gain curve(e.g., the gain curve shown in FIG. 11, or an adjusted version of thatgain curve which has been adjusted in response to feedback). FIG. 11 isa gain curve for channel distribution of content (e.g., PCM content) ofchannel Ls, with the vertical axis indicating gains, each of the gainsindicating a degree of distribution of each channel applied by therendering system (e.g., subsystem 803 of FIG. 3A, or subsystem 108 ofFIG. 3). For example, FIG. 11 indicates that content of channel Ls isrendered to channel Ls with full gain (“0” gain, as indicated by thevertical axis), that content of channel Ls is distributed to channel Ls2with a degree of distribution (indicated by the gain “−2” on thevertical axis), and that content of channel Ls is distributed to channelLrs1 with a lesser degree of distribution (approximately that indicatedby the gain “−4” on the vertical axis).

In response to the feedback (feedback 807 and/or 809), subsystem 803 maychange the degree of channel distribution of content of channel Ls. Thismay change each gradient of the FIG. 11 gain curve, e.g., by decreasing(making more negative) the slope of each segment of the gain curve,thereby reducing the amount of distribution of content of channel Ls toother speaker channels.

Examples of other changes in rendering (e.g., by subsystem 803 of FIG.3A or subsystem 108 of FIG. 3) which may be implemented in accordancewith some embodiments of the invention in response to feedback (e.g.,feedback 807 and/or feedback 809, or feedback 105) include thefollowing:

low priority object channels may be dropped (i.e.; not rendered) inresponse to the feedback;

the rendered size of at least one object, indicated by an object channel(e.g., an object channel of the output of subsystem 802 of FIG. 3A), maychange in response to compression feedback from a multiband limiter(e.g., limiter 806) or feedback from a smart amplifier (e.g., amplifier808). For example, if the limiter starts to limit, the rendered size ofan object could increase thus spreading the object energy to morespeakers; and/or

a method of bass extraction of at least one object, indicated by anobject channel (e.g., an object channel of the output of subsystem 802),may change in response to compression feedback from a multiband limiter(e.g., limiter 806) or feedback from a smart amplifier (e.g., amplifier808). For example, object bass extraction of two (or more) objectchannels may be controlled by the feedback in the same way that bassextraction is controlled in the above-described FIG. 4 embodiment (e.g.,bass content is extracted from a set of object channels and downmixed tomono, and then mixed back in to all or some of the channel outputs ofthe renderer), thus improving the power handling characteristics of thespeakers employed for playback (e.g., speakers of subsystem 109); and/or

the snapping of objects to speakers may change (e.g., its range may bechanged) in response to compression feedback from a multiband limiter(e.g., limiter 806) or feedback from a smart amplifier (e.g., amplifier808). If it starts to limit, the snapping of objects to speakers ispreferably gradually decreased to spread the audio to more speakers;and/or

the zone masking functionality may change (e.g., its range may bechanged) in response to compression feedback from a multiband limiter(e.g., limiter 806) or feedback from a smart amplifier (e.g., amplifier808), e.g., such that fewer zones are masked by the renderer when aglobal regulator (e.g., limiter 806) is limiting; and/or

the amount of trim that is applied by an object renderer to each objectmay change in response to compression feedback from a multiband limiter(e.g., limiter 806) or feedback from a smart amplifier (e.g., amplifier808). For example, it could be decreased as the amount of limitingincreases. This would give the effect of an increase in volume as thesignal level increases at the cost of spatial balance (since the back ofthe room would have more energy).

In some embodiments, dynamic range compression (DRC) gains and curvesthat are applied to an audio signal (e.g., by Dolby Digital Plus and AC4codecs) are controlled in response to compression feedback from amultiband limiter (e.g., limiter 104 of FIG. 3 or limiter 806 of FIG.3A) or feedback from a smart amplifier (e.g., feedback 809 fromamplifier 808). For example, when system 800 (or decoder subsystem 802and/or rendering subsystem 803) of FIG. 3A is implemented in or as adecoder (e.g., a codec), subsystem 802 (and/or subsystem 803) mayoperate in response to compression feedback 807 and/or feedback 809 toselectively apply DRC or to apply DRC more or less aggressively.Additionally, single ended DRC curves (e.g., those built into apost-processing library) may be applied (e.g., by decoder subsystem 802and/or rendering subsystem 803) more or less aggressively in response tocompression feedback from a multiband limiter (or feedback from a smartamplifier).

Some codecs (e.g., Dolby Digital Plus and AC4 codecs) and other decodershave end-point specific features, for example, speaker virtualization,headphone virtualization, and object audio versus speaker channel audio.In some embodiments, at least one such feature (e.g., virtualization) iscontrolled (e.g., within a decoder) in response to compression feedbackfrom a multiband limiter (e.g., limiter 104). For example, when system800 of FIG. 3A (or subsystem 100 of FIG. 3) is implemented in or as adecoder (e.g., codec), subsystem 102 (or subsystem 802 or 803 of FIG.3A) may operate in response to compression feedback (feedback 807 or105) or other feedback (e.g., feedback 809 from smart amplifier 808) toperform speaker virtualization, headphone virtualization, and/or otheraudio enhancement (and/or other processing) of object channels and/orspeaker channels.

In another example, a decoder (e.g., system 800 or subsystem 802 of FIG.3A) has both an object output mode and a speaker channel output mode(e.g., as does a Dolby Digital Plus decoder in the case of Joint ObjectCoding bitstreams), or is otherwise configured to selectively decode orignore individual object and/or speaker channels. If a multiband limiteris limiting, such a decoder (or a device including such a decoder) may(in response to compression feedback from the limiter) choose to operatein the speaker channel output mode of the decoder (or otherwise toignore all or some object channels, and instead to decode only speakerchannels) to save computational complexity (e.g., when the decoder is anelement of a mobile phone or other device having limited computationalcapabilities). Or, in response to compression feedback (e.g., feedback807) and/or feedback from a smart amplifier (e.g., feedback 809), such adecoder (or a device including such a decoder) may ignore all or someobject channels and instead decode only speaker channels (or otherwiseperform decoding in a manner determined at least in part by thecompression feedback and/or feedback from a smart amplifier).

Embodiments of the invention (e.g., those specifically described herein)can be combined to give superior performance and are not typicallymutually exclusive. In some such cases, smoothing (e.g., of compressionfeedback from a multiband limiter, or multiband limiter gain inputs)should be adjusted depending on the embodiments implemented.

In some embodiments, the invention is a system or device (e.g., aplayback device or other device having physically-limited or otherwiselimited bass reproduction capabilities, such as, for example, a notebookcomputer, laptop computer, tablet, soundbar, mobile phone, or otherdevice with at least one small speaker) configured to perform anyembodiment of the inventive method on an input audio signal. Forexample, the FIG. 3 system (or subsystem 100 and/or subsystem 108thereof) or the FIG. 3A system (or one or more subsystems thereof) maybe (or be included in): a playback device which includes all theelements of the FIG. 3 system (or of subsystem 100 and/or subsystem 108thereof) or the FIG. 3A system (or one or more subsystems thereof), sothat the device implements all such elements; or an audio processorwhich includes (in the sense that it implements) all of the elements ofthe FIG. 3 system (or subsystem 100 and/or subsystem 108 thereof) or theFIG. 3A system (or one or more subsystems thereof).

In a class of embodiments, the invention is an audio playback system(e.g., the FIG. 3 system (or subsystem 100 and/or subsystem 108thereof)) or the FIG. 3A system (or one or more subsystems thereof)implemented as a notebook, tablet, laptop, soundbar, mobile phone, orother device with small speakers, or a playback system which has limited(e.g., physically-limited) bass reproduction capabilities), and isconfigured to perform audio enhancement (e.g., bass enhancement) and/orrendering and/or decoding on audio in response to compression feedback(in accordance with any embodiment of the inventive method) and/orfeedback from a smart amplifier, to generate enhanced audio (e.g.,enhanced, compressed audio), and to render and play the enhanced audio.

In typical embodiments, the inventive system is or includes a general orspecial purpose processor (e.g., an implementation of subsystem 100and/or subsystem 108 of FIG. 3, or of the FIG. 3A system or one or moresubsystems thereof) programmed with software (or firmware) and/orotherwise configured to perform an embodiment of the inventive method.In some embodiments, the inventive system is a general purposeprocessor, coupled to receive input audio data, and programmed (withappropriate software) to generate output audio data in response to theinput audio data by performing an embodiment of the inventive method. Insome embodiments, the inventive system is a digital signal processor(e.g., an implementation subsystem 100 and/or subsystem 108 of FIG. 3 orthe FIG. 3A system or one or more subsystems thereof), coupled toreceive input audio data, and configured (e.g., programmed) to generateoutput audio data in response to the input audio data by performing anembodiment of the inventive method.

In some embodiments, the invention is a computer program product, thecomputer program product being tangibly stored on a non-transientcomputer-readable medium and comprising machine executable instructionswhich, when executed, cause the machine to perform any embodiment of theinventive method or steps thereof.

While specific embodiments of the present invention and applications ofthe invention have been described herein, it will be apparent to thoseof ordinary skill in the art that many variations on the embodiments andapplications described herein are possible without departing from thescope of the invention described and claimed herein. It should beunderstood that while certain forms of the invention have been shown anddescribed, the invention is not to be limited to the specificembodiments described and shown or the specific methods described.

1.-54. (canceled)
 55. A method for audio signal compression andenhancement, including: performing enhancement on a multichannel audiosignal to generate an enhanced multichannel audio signal; and performingmultiband compression on the enhanced multichannel audio signal using amultiband limiter, thereby generating a compressed, enhancedmultichannel audio signal, wherein the enhancement is performed inresponse to feedback from the multiband limiter, and wherein theenhancement comprises extracting a low frequency audio signal and a highfrequency audio signal from each channel of the multichannel audiosignal, summing together the extracted low frequency audio signals togenerate a summed signal, and mixing the summed signal back into eachextracted high frequency audio signal to generate the enhancedmultichannel audio signal.
 56. The method of claim 55, wherein thecompression is applied by two or more multiband limiters, each coupledand configured to generate compressed, enhanced audio for playback by adifferent subset of a set of speakers, and the enhancement includesdynamic routing of audio content of the multichannel audio signalbetween channels of the enhanced audio signal to be compressed bydifferent ones of the limiters.
 57. The method of claim 55, wherein theenhancement is performed in response to the compression feedback, andthe compression feedback is indicative of amount of compression appliedto each of at least two frequency bands of the enhanced audio signal.58. The method of claim 55, wherein the enhancement is performed inresponse to the compression feedback, and the compression feedback isindicative of at least one of amount or type of distortion predicted fora least one driver or to be prevented by the step of performingmultiband compression.
 59. The method of claim 55, wherein thecompression is applied by two or more multiband limiters, each coupledand configured to generate compressed, enhanced audio for playback by adifferent subset of a set of speakers, and the enhancement includesdynamic routing of audio content of the multichannel audio signalbetween channels of the enhanced audio signal to be compressed bydifferent ones of the limiters.
 60. The method of claim 55, wherein theenhancement includes at least one of bass volume enhancement, virtualbass enhancement, dialog enhancement, or virtualization.
 61. The methodof claim 55, wherein the multiband compression is performed on theenhanced multichannel audio signal, in a manner intended to preventdistortion upon playback of the compressed, enhanced multichannel audiosignal.
 62. A computer program product, the computer program productbeing tangibly stored on a non-transient computer-readable medium andcomprising machine executable instructions which, when executed, cause amachine to perform steps of the method of claim
 55. 63. A method foraudio signal decoding, including steps of: providing at least one offeedback from a smart amplifier or compression feedback; and decoding amultichannel audio signal in response to said at least one of thefeedback from the smart amplifier or the compression feedback, therebygenerating a decoded audio signal, wherein the compression feedback isindicative of at least one of: amount of compression applied or to beapplied to each of at least one frequency band of at least one channelof the decoded audio signal, or power or amplitude of the decoded audiosignal, or state of a system volume control.
 64. The method of claim 63,also including a step of: performing enhancement on the decoded audiosignal to generate an enhanced audio signal, wherein the enhancement isperformed in response to at least some of the feedback from the smartamplifier or at least some of the compression feedback.
 65. The methodof claim 63, wherein at least one of the decoding and the enhancement isperformed in response to the compression feedback, and the compressionfeedback is indicative of amount of compression applied to each of atleast two frequency bands of the decoded audio signal.
 66. The method ofclaim 63, wherein the multichannel audio signal includes objectchannels, and the decoding ignores, and does not decode, at least one ofthe object channels in response to said at least one of the feedbackfrom the smart amplifier or the compression feedback.
 67. The method ofclaim 63, wherein the decoding is performed in response to the feedbackfrom the smart amplifier, and said feedback is indicative of at leastone of temperature, voltage, current, impedance, or resistance, of atleast one speaker.
 68. A system for audio signal rendering, including: afeedback subsystem, coupled and configured to generate at least one ofsmart amplifier feedback or compression feedback; and a renderingsubsystem coupled and configured to render a multichannel audio inputsignal in response to said at least one of the smart amplifier feedbackand the compression feedback, thereby generating a multichannel audiosignal, wherein the compression feedback is indicative of at least oneof: amount of compression applied or to be applied to each of at leastone frequency band of the multichannel audio signal or an enhancedversion of the multichannel audio signal, or power or amplitude of themultichannel audio signal or the enhanced version of the multichannelaudio signal, or state of a system volume control.
 69. The system ofclaim 68, wherein the system also includes: an enhancement subsystem,coupled and configured to perform enhancement on the multichannel audiosignal to generate an enhanced audio signal, wherein the enhancementsubsystem is configured to perform the enhancement in response to atleast some of the smart amplifier feedback or at least some of thecompression feedback.
 70. The system of claim 68, wherein the enhancedaudio signal is an enhanced multichannel audio signal, and theenhancement includes dynamic routing of audio content of themultichannel audio signal between channels of the enhanced multichannelaudio signal.
 71. The system of claim 68, wherein the compressionfeedback is indicative of amount of compression applied to each of atleast two frequency bands of the multichannel audio signal or anenhanced version of the multichannel audio signal.
 72. The system ofclaim 68, wherein the compressed multichannel audio signal includesobject channels, and the rendering subsystem is configured to performobject-based audio rendering to render the multichannel audio inputsignal.
 73. The system of claim 68, wherein the multichannel audio inputsignal includes object channels, and the rendering subsystem isconfigured to ignore, and not to render, at least one of the objectchannels in response to said at least one of the smart amplifierfeedback or the compression feedback.
 74. The system of claim 68,wherein the rendering subsystem is configured to render the multichannelaudio input signal in response to the smart amplifier feedback, andwherein said smart amplifier feedback is indicative of at least one oftemperature, voltage, current, impedance, or resistance, of at least onespeaker.
 75. The system of claim 68, wherein said system is at least oneof: an audio playback system, a decoder system, a processor programmedto implement the rendering subsystem and the feedback subsystem and adigital signal processor programmed to implement the rendering subsystemand the feedback subsystem.